[==========] Running 17 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 6 tests from DiskFailure/DiskFailureITest
[ RUN ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/0
2026-08-10T22:03:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:03:59Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260810 22:03:59.370725 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.190:34181
--webserver_interface=127.27.17.190
--webserver_port=0
--builtin_ntp_servers=127.27.17.148:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.190:34181
--raft_prepare_replacement_before_eviction=false with env {}
W20260810 22:03:59.922752 27730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:03:59.923722 27730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:03:59.924228 27730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:03:59.955397 27730 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260810 22:03:59.955865 27730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:03:59.956161 27730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:03:59.956418 27730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:03:59.956668 27730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:03:59.992430 27730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.190:34181
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.190:34181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
--webserver_interface=127.27.17.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:03:59.994468 27730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:03:59.996651 27730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:00.049870 27739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:00.050127 27742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:00.052022 27740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:01.507342 27741 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1458 milliseconds
I20260810 22:04:01.507468 27730 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 22:04:01.508934 27730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:01.513921 27730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:01.515408 27730 hybrid_clock.cc:648] HybridClock initialized: now 1786399441515341 us; error 92 us; skew 500 ppm
I20260810 22:04:01.516538 27730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:01.527806 27730 webserver.cc:460] Webserver started at http://127.27.17.190:45969/ using document root <none> and password file <none>
I20260810 22:04:01.530861 27730 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:01.531106 27730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:04:01.531941 27730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:04:01.537575 27730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-0/instance:
uuid: "9a1244ce7efc411c91289f1ecd847644"
format_stamp: "Formatted at 2026-08-10 22:04:01 on dist-test-slave-bw88"
I20260810 22:04:01.538909 27730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-1/instance:
uuid: "9a1244ce7efc411c91289f1ecd847644"
format_stamp: "Formatted at 2026-08-10 22:04:01 on dist-test-slave-bw88"
I20260810 22:04:01.539815 27730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-2/instance:
uuid: "9a1244ce7efc411c91289f1ecd847644"
format_stamp: "Formatted at 2026-08-10 22:04:01 on dist-test-slave-bw88"
I20260810 22:04:01.540701 27730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-3/instance:
uuid: "9a1244ce7efc411c91289f1ecd847644"
format_stamp: "Formatted at 2026-08-10 22:04:01 on dist-test-slave-bw88"
I20260810 22:04:01.541738 27730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-4/instance:
uuid: "9a1244ce7efc411c91289f1ecd847644"
format_stamp: "Formatted at 2026-08-10 22:04:01 on dist-test-slave-bw88"
I20260810 22:04:01.542644 27730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/wal/instance:
uuid: "9a1244ce7efc411c91289f1ecd847644"
format_stamp: "Formatted at 2026-08-10 22:04:01 on dist-test-slave-bw88"
I20260810 22:04:01.564800 27730 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.017s sys 0.004s
I20260810 22:04:01.569828 27730 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:01.570091 27730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/wal
uuid: "9a1244ce7efc411c91289f1ecd847644"
format_stamp: "Formatted at 2026-08-10 22:04:01 on dist-test-slave-bw88"
I20260810 22:04:01.570506 27730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260810 22:04:01.611069 27730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:01.612736 27730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:01.613296 27730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:01.691884 27730 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.190:34181
I20260810 22:04:01.692353 27823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.190:34181 every 8 connection(s)
I20260810 22:04:01.695487 27730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
I20260810 22:04:01.700098 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 27730
I20260810 22:04:01.700838 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/wal/instance
I20260810 22:04:01.732978 27824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644: Bootstrap starting.
I20260810 22:04:01.740396 27824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:01.742803 27824 log.cc:824] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:01.748896 27824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644: No bootstrap required, opened a new log
I20260810 22:04:01.774381 27824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1244ce7efc411c91289f1ecd847644" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 34181 } }
I20260810 22:04:01.775178 27824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:01.775532 27824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1244ce7efc411c91289f1ecd847644, State: Initialized, Role: FOLLOWER
I20260810 22:04:01.776593 27824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1244ce7efc411c91289f1ecd847644" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 34181 } }
I20260810 22:04:01.777209 27824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:04:01.777609 27824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:04:01.777995 27824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:01.782608 27824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1244ce7efc411c91289f1ecd847644" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 34181 } }
I20260810 22:04:01.783602 27824 leader_election.cc:302] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a1244ce7efc411c91289f1ecd847644; no voters:
I20260810 22:04:01.785566 27824 leader_election.cc:288] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:04:01.786038 27832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:01.788615 27832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644 [term 1 LEADER]: Becoming Leader. State: Replica: 9a1244ce7efc411c91289f1ecd847644, State: Running, Role: LEADER
I20260810 22:04:01.789634 27832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1244ce7efc411c91289f1ecd847644" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 34181 } }
I20260810 22:04:01.790885 27824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:04:01.807742 27833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a1244ce7efc411c91289f1ecd847644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1244ce7efc411c91289f1ecd847644" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 34181 } } }
I20260810 22:04:01.808442 27833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644 [sys.catalog]: This master's current role is: LEADER
I20260810 22:04:01.809679 27834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a1244ce7efc411c91289f1ecd847644. Latest consensus state: current_term: 1 leader_uuid: "9a1244ce7efc411c91289f1ecd847644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1244ce7efc411c91289f1ecd847644" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 34181 } } }
I20260810 22:04:01.810178 27834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644 [sys.catalog]: This master's current role is: LEADER
I20260810 22:04:01.812892 27848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:04:01.829448 27848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:04:01.849982 27848 catalog_manager.cc:1347] Generated new cluster ID: 050c6b8d6d3c4539bcaf1206468f8fbd
I20260810 22:04:01.850330 27848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:04:01.864599 27848 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:04:01.866482 27848 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:04:01.882339 27848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a1244ce7efc411c91289f1ecd847644: Generated new TSK 0
I20260810 22:04:01.883419 27848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:04:01.899446 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.129:0
--local_ip_for_outbound_sockets=127.27.17.129
--webserver_interface=127.27.17.129
--webserver_port=0
--tserver_master_addrs=127.27.17.190:34181
--builtin_ntp_servers=127.27.17.148:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20260810 22:04:02.221694 27858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:02.222254 27858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:02.222776 27858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:02.254365 27858 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260810 22:04:02.254817 27858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:02.255638 27858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.129
I20260810 22:04:02.292079 27858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.27.17.129
--webserver_port=0
--tserver_master_addrs=127.27.17.190:34181
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.129
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:02.293742 27858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:04:02.295392 27858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:02.318607 27868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:02.319672 27872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:02.319191 27867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:03.990841 27869 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1669 milliseconds
I20260810 22:04:03.991082 27858 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 22:04:03.992524 27858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:03.995539 27858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:03.996920 27858 hybrid_clock.cc:648] HybridClock initialized: now 1786399443996897 us; error 79 us; skew 500 ppm
I20260810 22:04:03.997812 27858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:04.006807 27858 webserver.cc:460] Webserver started at http://127.27.17.129:42257/ using document root <none> and password file <none>
I20260810 22:04:04.008554 27858 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:04.008809 27858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:04:04.009555 27858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:04:04.014814 27858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/instance:
uuid: "8b0fbda82fb84730a663338f3552d36a"
format_stamp: "Formatted at 2026-08-10 22:04:04 on dist-test-slave-bw88"
I20260810 22:04:04.015996 27858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1/instance:
uuid: "8b0fbda82fb84730a663338f3552d36a"
format_stamp: "Formatted at 2026-08-10 22:04:04 on dist-test-slave-bw88"
I20260810 22:04:04.016914 27858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-2/instance:
uuid: "8b0fbda82fb84730a663338f3552d36a"
format_stamp: "Formatted at 2026-08-10 22:04:04 on dist-test-slave-bw88"
I20260810 22:04:04.018154 27858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-3/instance:
uuid: "8b0fbda82fb84730a663338f3552d36a"
format_stamp: "Formatted at 2026-08-10 22:04:04 on dist-test-slave-bw88"
I20260810 22:04:04.019052 27858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-4/instance:
uuid: "8b0fbda82fb84730a663338f3552d36a"
format_stamp: "Formatted at 2026-08-10 22:04:04 on dist-test-slave-bw88"
I20260810 22:04:04.020022 27858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/wal/instance:
uuid: "8b0fbda82fb84730a663338f3552d36a"
format_stamp: "Formatted at 2026-08-10 22:04:04 on dist-test-slave-bw88"
I20260810 22:04:04.048473 27858 fs_manager.cc:691] Time spent creating directory manager: real 0.028s user 0.014s sys 0.014s
I20260810 22:04:04.053790 27858 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:04.054095 27858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/wal
uuid: "8b0fbda82fb84730a663338f3552d36a"
format_stamp: "Formatted at 2026-08-10 22:04:04 on dist-test-slave-bw88"
I20260810 22:04:04.054529 27858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260810 22:04:04.098773 27858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:04.100739 27858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:04.101447 27858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:04.105664 27858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:04:04.112010 27858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:04:04.112316 27858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:04.112599 27858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:04:04.112792 27858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:04.302668 27858 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.129:41487
I20260810 22:04:04.303259 28039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.129:41487 every 8 connection(s)
I20260810 22:04:04.305439 27858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
I20260810 22:04:04.309855 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 27858
I20260810 22:04:04.310348 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/wal/instance
I20260810 22:04:04.318421 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.130:0
--local_ip_for_outbound_sockets=127.27.17.130
--webserver_interface=127.27.17.130
--webserver_port=0
--tserver_master_addrs=127.27.17.190:34181
--builtin_ntp_servers=127.27.17.148:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20260810 22:04:04.335948 28040 heartbeater.cc:347] Connected to a master server at 127.27.17.190:34181
I20260810 22:04:04.336555 28040 heartbeater.cc:464] Registering TS with master...
I20260810 22:04:04.338078 28040 heartbeater.cc:511] Master 127.27.17.190:34181 requested a full tablet report, sending...
I20260810 22:04:04.341918 27780 ts_manager.cc:194] Registered new tserver with Master: 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487)
I20260810 22:04:04.344412 27780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.129:55705
W20260810 22:04:04.658792 28044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:04.659355 28044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:04.659876 28044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:04.690804 28044 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260810 22:04:04.691308 28044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:04.692307 28044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.130
I20260810 22:04:04.727313 28044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
--webserver_interface=127.27.17.130
--webserver_port=0
--tserver_master_addrs=127.27.17.190:34181
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.130
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:04.728598 28044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:04:04.730197 28044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:04.750572 28052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:05.349644 28040 heartbeater.cc:503] Master 127.27.17.190:34181 was elected leader, sending a full tablet report...
W20260810 22:04:04.751062 28054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:04.752463 28044 server_base.cc:1034] running on GCE node
W20260810 22:04:04.751331 28051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:05.954308 28044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:05.956647 28044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:05.958024 28044 hybrid_clock.cc:648] HybridClock initialized: now 1786399445957981 us; error 50 us; skew 500 ppm
I20260810 22:04:05.958792 28044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:05.965816 28044 webserver.cc:460] Webserver started at http://127.27.17.130:32847/ using document root <none> and password file <none>
I20260810 22:04:05.967247 28044 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:05.967437 28044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:04:05.968072 28044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:04:05.973333 28044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0/instance:
uuid: "9c63787081744750872196ba0013abed"
format_stamp: "Formatted at 2026-08-10 22:04:05 on dist-test-slave-bw88"
I20260810 22:04:05.974471 28044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-1/instance:
uuid: "9c63787081744750872196ba0013abed"
format_stamp: "Formatted at 2026-08-10 22:04:05 on dist-test-slave-bw88"
I20260810 22:04:05.975381 28044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-2/instance:
uuid: "9c63787081744750872196ba0013abed"
format_stamp: "Formatted at 2026-08-10 22:04:05 on dist-test-slave-bw88"
I20260810 22:04:05.976227 28044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-3/instance:
uuid: "9c63787081744750872196ba0013abed"
format_stamp: "Formatted at 2026-08-10 22:04:05 on dist-test-slave-bw88"
I20260810 22:04:05.977126 28044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-4/instance:
uuid: "9c63787081744750872196ba0013abed"
format_stamp: "Formatted at 2026-08-10 22:04:05 on dist-test-slave-bw88"
I20260810 22:04:05.978111 28044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/wal/instance:
uuid: "9c63787081744750872196ba0013abed"
format_stamp: "Formatted at 2026-08-10 22:04:05 on dist-test-slave-bw88"
I20260810 22:04:06.000483 28044 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.015s sys 0.006s
I20260810 22:04:06.005043 28044 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:06.005277 28044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/wal
uuid: "9c63787081744750872196ba0013abed"
format_stamp: "Formatted at 2026-08-10 22:04:05 on dist-test-slave-bw88"
I20260810 22:04:06.005687 28044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260810 22:04:06.054824 28044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:06.056404 28044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:06.056835 28044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:06.059719 28044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:04:06.064630 28044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:04:06.064867 28044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:06.065162 28044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:04:06.065307 28044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:06.221477 28044 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.130:37287
I20260810 22:04:06.222034 28219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.130:37287 every 8 connection(s)
I20260810 22:04:06.224175 28044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
I20260810 22:04:06.226804 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 28044
I20260810 22:04:06.227336 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/wal/instance
I20260810 22:04:06.235191 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.131:0
--local_ip_for_outbound_sockets=127.27.17.131
--webserver_interface=127.27.17.131
--webserver_port=0
--tserver_master_addrs=127.27.17.190:34181
--builtin_ntp_servers=127.27.17.148:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20260810 22:04:06.251001 28220 heartbeater.cc:347] Connected to a master server at 127.27.17.190:34181
I20260810 22:04:06.251546 28220 heartbeater.cc:464] Registering TS with master...
I20260810 22:04:06.252652 28220 heartbeater.cc:511] Master 127.27.17.190:34181 requested a full tablet report, sending...
I20260810 22:04:06.255103 27780 ts_manager.cc:194] Registered new tserver with Master: 9c63787081744750872196ba0013abed (127.27.17.130:37287)
I20260810 22:04:06.257397 27780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.130:36397
W20260810 22:04:06.566174 28227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:06.566744 28227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:06.567261 28227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:06.599613 28227 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260810 22:04:06.600030 28227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:06.600868 28227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.131
I20260810 22:04:06.635864 28227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
--webserver_interface=127.27.17.131
--webserver_port=0
--tserver_master_addrs=127.27.17.190:34181
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.131
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:06.637320 28227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:04:06.639014 28227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:06.658165 28236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:07.261587 28220 heartbeater.cc:503] Master 127.27.17.190:34181 was elected leader, sending a full tablet report...
W20260810 22:04:06.658259 28237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:06.660283 28239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:07.860024 28238 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 22:04:07.860139 28227 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 22:04:07.864082 28227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:07.866783 28227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:07.868212 28227 hybrid_clock.cc:648] HybridClock initialized: now 1786399447868179 us; error 69 us; skew 500 ppm
I20260810 22:04:07.869100 28227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:07.880100 28227 webserver.cc:460] Webserver started at http://127.27.17.131:44241/ using document root <none> and password file <none>
I20260810 22:04:07.881706 28227 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:07.881969 28227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:04:07.882679 28227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:04:07.887737 28227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0/instance:
uuid: "050f5ddd2d7c44c8a923f9f876d4e186"
format_stamp: "Formatted at 2026-08-10 22:04:07 on dist-test-slave-bw88"
I20260810 22:04:07.888939 28227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-1/instance:
uuid: "050f5ddd2d7c44c8a923f9f876d4e186"
format_stamp: "Formatted at 2026-08-10 22:04:07 on dist-test-slave-bw88"
I20260810 22:04:07.889997 28227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-2/instance:
uuid: "050f5ddd2d7c44c8a923f9f876d4e186"
format_stamp: "Formatted at 2026-08-10 22:04:07 on dist-test-slave-bw88"
I20260810 22:04:07.890919 28227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-3/instance:
uuid: "050f5ddd2d7c44c8a923f9f876d4e186"
format_stamp: "Formatted at 2026-08-10 22:04:07 on dist-test-slave-bw88"
I20260810 22:04:07.891909 28227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-4/instance:
uuid: "050f5ddd2d7c44c8a923f9f876d4e186"
format_stamp: "Formatted at 2026-08-10 22:04:07 on dist-test-slave-bw88"
I20260810 22:04:07.892915 28227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/wal/instance:
uuid: "050f5ddd2d7c44c8a923f9f876d4e186"
format_stamp: "Formatted at 2026-08-10 22:04:07 on dist-test-slave-bw88"
I20260810 22:04:07.919239 28227 fs_manager.cc:691] Time spent creating directory manager: real 0.026s user 0.019s sys 0.005s
I20260810 22:04:07.924199 28227 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:07.924479 28227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/wal
uuid: "050f5ddd2d7c44c8a923f9f876d4e186"
format_stamp: "Formatted at 2026-08-10 22:04:07 on dist-test-slave-bw88"
I20260810 22:04:07.925034 28227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260810 22:04:07.972083 28227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:07.973722 28227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:07.974191 28227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:07.978266 28227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:04:07.983402 28227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:04:07.983657 28227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:07.983937 28227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:04:07.984119 28227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:08.143982 28227 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.131:36895
I20260810 22:04:08.144500 28421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.131:36895 every 8 connection(s)
I20260810 22:04:08.146838 28227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
I20260810 22:04:08.147747 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 28227
I20260810 22:04:08.148169 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/wal/instance
I20260810 22:04:08.173398 28422 heartbeater.cc:347] Connected to a master server at 127.27.17.190:34181
I20260810 22:04:08.173851 28422 heartbeater.cc:464] Registering TS with master...
I20260810 22:04:08.174808 28422 heartbeater.cc:511] Master 127.27.17.190:34181 requested a full tablet report, sending...
I20260810 22:04:08.176925 27780 ts_manager.cc:194] Registered new tserver with Master: 050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131:36895)
I20260810 22:04:08.178445 27780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.131:59895
I20260810 22:04:08.184540 27718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:04:08.225118 27718 test_util.cc:268] Using random seed: -2004232899
I20260810 22:04:08.277455 27780 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60846:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\231\231\231\031\004\001\0002333\004\001\000\313\314\314L\004\001\000dfff"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 22:04:08.355334 27950 tablet_service.cc:1467] Processing CreateTablet for tablet 140448518535465eafa77555fa59db8c (DEFAULT_TABLE table=test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260810 22:04:08.357537 27953 tablet_service.cc:1467] Processing CreateTablet for tablet 312e5bba4bc240d39b6d15652a40cc79 (DEFAULT_TABLE table=test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260810 22:04:08.358022 27955 tablet_service.cc:1467] Processing CreateTablet for tablet 19b09a379fab4ad6aa1985c84fe6654a (DEFAULT_TABLE table=test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260810 22:04:08.355954 27952 tablet_service.cc:1467] Processing CreateTablet for tablet c849b79d77074d9eb6cfe7699ac8e78e (DEFAULT_TABLE table=test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260810 22:04:08.356864 27951 tablet_service.cc:1467] Processing CreateTablet for tablet b5bd5361adba43f09bdf5c51cda4bbcd (DEFAULT_TABLE table=test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260810 22:04:08.373176 28319 tablet_service.cc:1467] Processing CreateTablet for tablet c849b79d77074d9eb6cfe7699ac8e78e (DEFAULT_TABLE table=test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260810 22:04:08.374483 28134 tablet_service.cc:1467] Processing CreateTablet for tablet 312e5bba4bc240d39b6d15652a40cc79 (DEFAULT_TABLE table=test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260810 22:04:08.378074 28318 tablet_service.cc:1467] Processing CreateTablet for tablet b5bd5361adba43f09bdf5c51cda4bbcd (DEFAULT_TABLE table=test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260810 22:04:08.371197 28320 tablet_service.cc:1467] Processing CreateTablet for tablet 312e5bba4bc240d39b6d15652a40cc79 (DEFAULT_TABLE table=test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260810 22:04:08.385536 28135 tablet_service.cc:1467] Processing CreateTablet for tablet 19b09a379fab4ad6aa1985c84fe6654a (DEFAULT_TABLE table=test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260810 22:04:08.386302 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 19b09a379fab4ad6aa1985c84fe6654a (DEFAULT_TABLE table=test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260810 22:04:08.388383 28133 tablet_service.cc:1467] Processing CreateTablet for tablet c849b79d77074d9eb6cfe7699ac8e78e (DEFAULT_TABLE table=test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260810 22:04:08.390234 28132 tablet_service.cc:1467] Processing CreateTablet for tablet b5bd5361adba43f09bdf5c51cda4bbcd (DEFAULT_TABLE table=test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260810 22:04:08.396262 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 140448518535465eafa77555fa59db8c (DEFAULT_TABLE table=test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260810 22:04:08.400421 28317 tablet_service.cc:1467] Processing CreateTablet for tablet 140448518535465eafa77555fa59db8c (DEFAULT_TABLE table=test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260810 22:04:08.420197 28452 tablet_bootstrap.cc:492] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: Bootstrap starting.
I20260810 22:04:08.426523 28455 tablet_bootstrap.cc:492] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: Bootstrap starting.
I20260810 22:04:08.439663 28451 tablet_bootstrap.cc:492] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a: Bootstrap starting.
I20260810 22:04:08.441593 28457 tablet_bootstrap.cc:492] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: Bootstrap starting.
I20260810 22:04:08.451305 28452 tablet_bootstrap.cc:654] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:08.453133 28455 tablet_bootstrap.cc:654] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:08.454056 28452 log.cc:824] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:08.453389 28451 tablet_bootstrap.cc:654] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:08.453863 28456 tablet_bootstrap.cc:492] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a: Bootstrap starting.
I20260810 22:04:08.468276 28457 tablet_bootstrap.cc:654] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:08.468709 28459 tablet_bootstrap.cc:492] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186: Bootstrap starting.
I20260810 22:04:08.475301 28456 tablet_bootstrap.cc:654] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:08.484179 28460 tablet_bootstrap.cc:492] T 19b09a379fab4ad6aa1985c84fe6654a P 050f5ddd2d7c44c8a923f9f876d4e186: Bootstrap starting.
I20260810 22:04:08.503193 28451 tablet_bootstrap.cc:492] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a: No bootstrap required, opened a new log
I20260810 22:04:08.504805 28451 ts_tablet_manager.cc:1397] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.008s
I20260810 22:04:08.504133 28455 tablet_bootstrap.cc:492] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: No bootstrap required, opened a new log
I20260810 22:04:08.505686 28455 ts_tablet_manager.cc:1397] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: Time spent bootstrapping tablet: real 0.080s user 0.012s sys 0.007s
I20260810 22:04:08.514703 28452 tablet_bootstrap.cc:492] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: No bootstrap required, opened a new log
I20260810 22:04:08.515230 28471 tablet_bootstrap.cc:492] T 140448518535465eafa77555fa59db8c P 9c63787081744750872196ba0013abed: Bootstrap starting.
I20260810 22:04:08.515242 28452 ts_tablet_manager.cc:1397] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: Time spent bootstrapping tablet: real 0.096s user 0.020s sys 0.043s
I20260810 22:04:08.519762 28464 tablet_bootstrap.cc:492] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186: Bootstrap starting.
I20260810 22:04:08.521570 28467 tablet_bootstrap.cc:492] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed: Bootstrap starting.
I20260810 22:04:08.522265 28469 tablet_bootstrap.cc:492] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed: Bootstrap starting.
I20260810 22:04:08.531441 28457 tablet_bootstrap.cc:492] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: No bootstrap required, opened a new log
I20260810 22:04:08.532392 28457 ts_tablet_manager.cc:1397] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: Time spent bootstrapping tablet: real 0.091s user 0.025s sys 0.013s
I20260810 22:04:08.542248 28456 tablet_bootstrap.cc:492] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a: No bootstrap required, opened a new log
I20260810 22:04:08.542692 28456 ts_tablet_manager.cc:1397] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a: Time spent bootstrapping tablet: real 0.096s user 0.009s sys 0.003s
I20260810 22:04:08.546979 28467 tablet_bootstrap.cc:654] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:08.551137 28475 tablet_bootstrap.cc:492] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186: Bootstrap starting.
I20260810 22:04:08.554250 28472 tablet_bootstrap.cc:492] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed: Bootstrap starting.
I20260810 22:04:08.559725 28455 raft_consensus.cc:348] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.560757 28455 raft_consensus.cc:374] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:08.555810 28470 tablet_bootstrap.cc:492] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed: Bootstrap starting.
I20260810 22:04:08.562188 28475 tablet_bootstrap.cc:654] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:08.561760 28452 raft_consensus.cc:348] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.562521 28452 raft_consensus.cc:374] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:08.562863 28452 raft_consensus.cc:729] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0fbda82fb84730a663338f3552d36a, State: Initialized, Role: FOLLOWER
I20260810 22:04:08.563145 28464 tablet_bootstrap.cc:654] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:08.563839 28452 consensus_queue.cc:260] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.564692 28475 log.cc:824] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:08.559526 28469 tablet_bootstrap.cc:654] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:08.554677 28471 tablet_bootstrap.cc:654] T 140448518535465eafa77555fa59db8c P 9c63787081744750872196ba0013abed: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:08.565922 28451 raft_consensus.cc:348] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.566735 28451 raft_consensus.cc:374] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:08.567011 28460 tablet_bootstrap.cc:654] T 19b09a379fab4ad6aa1985c84fe6654a P 050f5ddd2d7c44c8a923f9f876d4e186: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:08.567009 28451 raft_consensus.cc:729] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0fbda82fb84730a663338f3552d36a, State: Initialized, Role: FOLLOWER
I20260810 22:04:08.567301 28473 tablet_bootstrap.cc:492] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186: Bootstrap starting.
I20260810 22:04:08.561097 28455 raft_consensus.cc:729] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0fbda82fb84730a663338f3552d36a, State: Initialized, Role: FOLLOWER
I20260810 22:04:08.568053 28455 consensus_queue.cc:260] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.570258 28459 tablet_bootstrap.cc:654] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:08.571266 28451 consensus_queue.cc:260] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.570181 28467 log.cc:824] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:08.576342 28455 ts_tablet_manager.cc:1428] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: Time spent starting tablet: real 0.070s user 0.015s sys 0.007s
I20260810 22:04:08.578270 28452 ts_tablet_manager.cc:1428] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: Time spent starting tablet: real 0.063s user 0.006s sys 0.004s
I20260810 22:04:08.579213 28457 raft_consensus.cc:348] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.580040 28457 raft_consensus.cc:374] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:08.580351 28457 raft_consensus.cc:729] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0fbda82fb84730a663338f3552d36a, State: Initialized, Role: FOLLOWER
I20260810 22:04:08.581403 28456 raft_consensus.cc:348] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.583199 28457 consensus_queue.cc:260] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.586851 28473 tablet_bootstrap.cc:654] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:08.589468 28451 ts_tablet_manager.cc:1428] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a: Time spent starting tablet: real 0.084s user 0.012s sys 0.008s
I20260810 22:04:08.582218 28456 raft_consensus.cc:374] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:08.600725 28456 raft_consensus.cc:729] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0fbda82fb84730a663338f3552d36a, State: Initialized, Role: FOLLOWER
I20260810 22:04:08.601578 28456 consensus_queue.cc:260] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.602389 28457 ts_tablet_manager.cc:1428] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: Time spent starting tablet: real 0.070s user 0.016s sys 0.001s
I20260810 22:04:08.603871 28456 ts_tablet_manager.cc:1428] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a: Time spent starting tablet: real 0.061s user 0.007s sys 0.000s
I20260810 22:04:08.604414 28472 tablet_bootstrap.cc:654] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:08.608402 28470 tablet_bootstrap.cc:654] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:08.621038 28459 tablet_bootstrap.cc:492] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186: No bootstrap required, opened a new log
I20260810 22:04:08.621848 28459 ts_tablet_manager.cc:1397] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186: Time spent bootstrapping tablet: real 0.154s user 0.011s sys 0.011s
I20260810 22:04:08.609829 28464 tablet_bootstrap.cc:492] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186: No bootstrap required, opened a new log
I20260810 22:04:08.623683 28464 ts_tablet_manager.cc:1397] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186: Time spent bootstrapping tablet: real 0.104s user 0.006s sys 0.017s
I20260810 22:04:08.625860 28469 tablet_bootstrap.cc:492] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed: No bootstrap required, opened a new log
I20260810 22:04:08.626592 28469 ts_tablet_manager.cc:1397] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed: Time spent bootstrapping tablet: real 0.105s user 0.011s sys 0.010s
I20260810 22:04:08.628783 28467 tablet_bootstrap.cc:492] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed: No bootstrap required, opened a new log
I20260810 22:04:08.633257 28479 raft_consensus.cc:482] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:08.634034 28479 raft_consensus.cc:504] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.635721 28460 tablet_bootstrap.cc:492] T 19b09a379fab4ad6aa1985c84fe6654a P 050f5ddd2d7c44c8a923f9f876d4e186: No bootstrap required, opened a new log
I20260810 22:04:08.636272 28460 ts_tablet_manager.cc:1397] T 19b09a379fab4ad6aa1985c84fe6654a P 050f5ddd2d7c44c8a923f9f876d4e186: Time spent bootstrapping tablet: real 0.152s user 0.009s sys 0.010s
I20260810 22:04:08.638067 28479 leader_election.cc:288] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c63787081744750872196ba0013abed (127.27.17.130:37287), 050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131:36895)
I20260810 22:04:08.617689 28475 tablet_bootstrap.cc:492] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186: No bootstrap required, opened a new log
I20260810 22:04:08.640755 28467 ts_tablet_manager.cc:1397] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed: Time spent bootstrapping tablet: real 0.120s user 0.018s sys 0.006s
I20260810 22:04:08.640643 28475 ts_tablet_manager.cc:1397] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186: Time spent bootstrapping tablet: real 0.090s user 0.017s sys 0.015s
I20260810 22:04:08.645015 28471 tablet_bootstrap.cc:492] T 140448518535465eafa77555fa59db8c P 9c63787081744750872196ba0013abed: No bootstrap required, opened a new log
I20260810 22:04:08.645707 28471 ts_tablet_manager.cc:1397] T 140448518535465eafa77555fa59db8c P 9c63787081744750872196ba0013abed: Time spent bootstrapping tablet: real 0.131s user 0.018s sys 0.004s
I20260810 22:04:08.649922 28470 tablet_bootstrap.cc:492] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed: No bootstrap required, opened a new log
I20260810 22:04:08.650446 28470 ts_tablet_manager.cc:1397] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed: Time spent bootstrapping tablet: real 0.095s user 0.008s sys 0.007s
I20260810 22:04:08.656649 28473 tablet_bootstrap.cc:492] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186: No bootstrap required, opened a new log
I20260810 22:04:08.657104 28473 ts_tablet_manager.cc:1397] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186: Time spent bootstrapping tablet: real 0.090s user 0.012s sys 0.008s
I20260810 22:04:08.664072 28472 tablet_bootstrap.cc:492] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed: No bootstrap required, opened a new log
I20260810 22:04:08.664517 28472 ts_tablet_manager.cc:1397] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed: Time spent bootstrapping tablet: real 0.111s user 0.009s sys 0.035s
I20260810 22:04:08.674808 28460 raft_consensus.cc:348] T 19b09a379fab4ad6aa1985c84fe6654a P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.675823 28460 raft_consensus.cc:374] T 19b09a379fab4ad6aa1985c84fe6654a P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:08.676149 28460 raft_consensus.cc:729] T 19b09a379fab4ad6aa1985c84fe6654a P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 050f5ddd2d7c44c8a923f9f876d4e186, State: Initialized, Role: FOLLOWER
I20260810 22:04:08.677098 28460 consensus_queue.cc:260] T 19b09a379fab4ad6aa1985c84fe6654a P 050f5ddd2d7c44c8a923f9f876d4e186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.677441 28459 raft_consensus.cc:348] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.677800 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c849b79d77074d9eb6cfe7699ac8e78e" candidate_uuid: "8b0fbda82fb84730a663338f3552d36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" is_pre_election: true
I20260810 22:04:08.678166 28459 raft_consensus.cc:374] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:08.678544 28459 raft_consensus.cc:729] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 050f5ddd2d7c44c8a923f9f876d4e186, State: Initialized, Role: FOLLOWER
I20260810 22:04:08.679445 28459 consensus_queue.cc:260] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.680941 28467 raft_consensus.cc:348] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.681469 28158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c849b79d77074d9eb6cfe7699ac8e78e" candidate_uuid: "8b0fbda82fb84730a663338f3552d36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c63787081744750872196ba0013abed" is_pre_election: true
I20260810 22:04:08.682049 28467 raft_consensus.cc:374] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260810 22:04:08.681826 27886 leader_election.cc:341] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131:36895): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:04:08.681950 28473 raft_consensus.cc:348] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.682369 28467 raft_consensus.cc:729] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c63787081744750872196ba0013abed, State: Initialized, Role: FOLLOWER
I20260810 22:04:08.682691 28473 raft_consensus.cc:374] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:08.683022 28473 raft_consensus.cc:729] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 050f5ddd2d7c44c8a923f9f876d4e186, State: Initialized, Role: FOLLOWER
I20260810 22:04:08.683331 28470 raft_consensus.cc:348] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.683348 28467 consensus_queue.cc:260] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.684116 28470 raft_consensus.cc:374] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:08.684407 28470 raft_consensus.cc:729] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c63787081744750872196ba0013abed, State: Initialized, Role: FOLLOWER
I20260810 22:04:08.684670 28158 raft_consensus.cc:2454] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b0fbda82fb84730a663338f3552d36a in term 0.
I20260810 22:04:08.684464 28473 consensus_queue.cc:260] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.685144 28470 consensus_queue.cc:260] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.688828 28475 raft_consensus.cc:348] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.689033 28472 raft_consensus.cc:348] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.690086 28472 raft_consensus.cc:374] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:08.689545 28475 raft_consensus.cc:374] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:08.692797 28475 raft_consensus.cc:729] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 050f5ddd2d7c44c8a923f9f876d4e186, State: Initialized, Role: FOLLOWER
I20260810 22:04:08.691095 28471 raft_consensus.cc:348] T 140448518535465eafa77555fa59db8c P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.694151 28471 raft_consensus.cc:374] T 140448518535465eafa77555fa59db8c P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:08.694540 28471 raft_consensus.cc:729] T 140448518535465eafa77555fa59db8c P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c63787081744750872196ba0013abed, State: Initialized, Role: FOLLOWER
I20260810 22:04:08.694387 28464 raft_consensus.cc:348] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.695113 28464 raft_consensus.cc:374] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:08.690435 28472 raft_consensus.cc:729] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c63787081744750872196ba0013abed, State: Initialized, Role: FOLLOWER
I20260810 22:04:08.695372 28464 raft_consensus.cc:729] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 050f5ddd2d7c44c8a923f9f876d4e186, State: Initialized, Role: FOLLOWER
I20260810 22:04:08.696065 28472 consensus_queue.cc:260] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.696626 28475 consensus_queue.cc:260] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.699399 28469 raft_consensus.cc:348] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.700326 28469 raft_consensus.cc:374] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:08.700651 28469 raft_consensus.cc:729] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c63787081744750872196ba0013abed, State: Initialized, Role: FOLLOWER
I20260810 22:04:08.702220 28464 consensus_queue.cc:260] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.702785 28471 consensus_queue.cc:260] T 140448518535465eafa77555fa59db8c P 9c63787081744750872196ba0013abed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.713503 27888 leader_election.cc:302] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8b0fbda82fb84730a663338f3552d36a, 9c63787081744750872196ba0013abed; no voters: 050f5ddd2d7c44c8a923f9f876d4e186
I20260810 22:04:08.714928 28479 raft_consensus.cc:2781] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:08.715382 28479 raft_consensus.cc:482] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:08.715741 28459 ts_tablet_manager.cc:1428] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186: Time spent starting tablet: real 0.094s user 0.027s sys 0.005s
I20260810 22:04:08.715785 28479 raft_consensus.cc:3037] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:08.715277 28469 consensus_queue.cc:260] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.720674 28470 ts_tablet_manager.cc:1428] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed: Time spent starting tablet: real 0.070s user 0.003s sys 0.013s
I20260810 22:04:08.720810 28422 heartbeater.cc:503] Master 127.27.17.190:34181 was elected leader, sending a full tablet report...
I20260810 22:04:08.726441 28475 ts_tablet_manager.cc:1428] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186: Time spent starting tablet: real 0.084s user 0.018s sys 0.001s
I20260810 22:04:08.724852 28472 ts_tablet_manager.cc:1428] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed: Time spent starting tablet: real 0.060s user 0.006s sys 0.008s
I20260810 22:04:08.729223 28471 ts_tablet_manager.cc:1428] T 140448518535465eafa77555fa59db8c P 9c63787081744750872196ba0013abed: Time spent starting tablet: real 0.083s user 0.012s sys 0.008s
I20260810 22:04:08.729000 28473 ts_tablet_manager.cc:1428] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186: Time spent starting tablet: real 0.072s user 0.005s sys 0.011s
I20260810 22:04:08.731527 28464 ts_tablet_manager.cc:1428] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186: Time spent starting tablet: real 0.107s user 0.015s sys 0.005s
W20260810 22:04:08.732795 28224 tablet.cc:2369] T 140448518535465eafa77555fa59db8c P 9c63787081744750872196ba0013abed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:04:08.740154 28479 raft_consensus.cc:504] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.742666 28467 ts_tablet_manager.cc:1428] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed: Time spent starting tablet: real 0.101s user 0.015s sys 0.020s
I20260810 22:04:08.743088 28460 ts_tablet_manager.cc:1428] T 19b09a379fab4ad6aa1985c84fe6654a P 050f5ddd2d7c44c8a923f9f876d4e186: Time spent starting tablet: real 0.106s user 0.004s sys 0.017s
I20260810 22:04:08.745934 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c849b79d77074d9eb6cfe7699ac8e78e" candidate_uuid: "8b0fbda82fb84730a663338f3552d36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "050f5ddd2d7c44c8a923f9f876d4e186"
I20260810 22:04:08.746667 28352 raft_consensus.cc:3037] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:08.755059 28352 raft_consensus.cc:2454] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b0fbda82fb84730a663338f3552d36a in term 1.
I20260810 22:04:08.745863 28158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c849b79d77074d9eb6cfe7699ac8e78e" candidate_uuid: "8b0fbda82fb84730a663338f3552d36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c63787081744750872196ba0013abed"
I20260810 22:04:08.755817 28158 raft_consensus.cc:3037] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:08.763744 28158 raft_consensus.cc:2454] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b0fbda82fb84730a663338f3552d36a in term 1.
I20260810 22:04:08.765529 27888 leader_election.cc:302] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b0fbda82fb84730a663338f3552d36a, 9c63787081744750872196ba0013abed; no voters:
I20260810 22:04:08.765807 28505 raft_consensus.cc:482] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:08.766388 28479 leader_election.cc:288] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [CANDIDATE]: Term 1 election: Requested vote from peers 9c63787081744750872196ba0013abed (127.27.17.130:37287), 050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131:36895)
I20260810 22:04:08.766297 28505 raft_consensus.cc:504] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.767167 28479 raft_consensus.cc:2781] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:08.767699 28479 raft_consensus.cc:686] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [term 1 LEADER]: Becoming Leader. State: Replica: 8b0fbda82fb84730a663338f3552d36a, State: Running, Role: LEADER
I20260810 22:04:08.769798 28505 leader_election.cc:288] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487), 050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131:36895)
I20260810 22:04:08.770931 28479 consensus_queue.cc:237] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.776918 28514 raft_consensus.cc:482] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:08.782092 28514 raft_consensus.cc:504] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.784552 28514 leader_election.cc:288] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487), 050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131:36895)
I20260810 22:04:08.786082 28469 ts_tablet_manager.cc:1428] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed: Time spent starting tablet: real 0.159s user 0.036s sys 0.024s
I20260810 22:04:08.786224 27779 catalog_manager.cc:5495] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b0fbda82fb84730a663338f3552d36a (127.27.17.129). New cstate: current_term: 1 leader_uuid: "8b0fbda82fb84730a663338f3552d36a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:08.791988 28479 raft_consensus.cc:482] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:08.792459 28479 raft_consensus.cc:504] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.794229 28479 leader_election.cc:288] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c63787081744750872196ba0013abed (127.27.17.130:37287), 050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131:36895)
I20260810 22:04:08.795763 28158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "312e5bba4bc240d39b6d15652a40cc79" candidate_uuid: "8b0fbda82fb84730a663338f3552d36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c63787081744750872196ba0013abed" is_pre_election: true
I20260810 22:04:08.796375 28158 raft_consensus.cc:2454] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b0fbda82fb84730a663338f3552d36a in term 0.
I20260810 22:04:08.796612 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "312e5bba4bc240d39b6d15652a40cc79" candidate_uuid: "8b0fbda82fb84730a663338f3552d36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" is_pre_election: true
I20260810 22:04:08.797278 28352 raft_consensus.cc:2454] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b0fbda82fb84730a663338f3552d36a in term 0.
I20260810 22:04:08.797550 27888 leader_election.cc:302] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b0fbda82fb84730a663338f3552d36a, 9c63787081744750872196ba0013abed; no voters:
I20260810 22:04:08.798336 28479 raft_consensus.cc:2781] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:08.798699 28479 raft_consensus.cc:482] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:08.799089 28479 raft_consensus.cc:3037] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:08.805540 28479 raft_consensus.cc:504] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.808262 28158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "312e5bba4bc240d39b6d15652a40cc79" candidate_uuid: "8b0fbda82fb84730a663338f3552d36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c63787081744750872196ba0013abed"
I20260810 22:04:08.808852 28158 raft_consensus.cc:3037] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:08.810869 27982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b09a379fab4ad6aa1985c84fe6654a" candidate_uuid: "9c63787081744750872196ba0013abed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0fbda82fb84730a663338f3552d36a" is_pre_election: true
I20260810 22:04:08.811755 27982 raft_consensus.cc:2454] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c63787081744750872196ba0013abed in term 0.
I20260810 22:04:08.813177 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "312e5bba4bc240d39b6d15652a40cc79" candidate_uuid: "9c63787081744750872196ba0013abed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0fbda82fb84730a663338f3552d36a" is_pre_election: true
I20260810 22:04:08.813355 28075 leader_election.cc:302] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b0fbda82fb84730a663338f3552d36a, 9c63787081744750872196ba0013abed; no voters:
I20260810 22:04:08.814312 27981 raft_consensus.cc:2379] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c63787081744750872196ba0013abed in current term 1: Already voted for candidate 8b0fbda82fb84730a663338f3552d36a in this term.
I20260810 22:04:08.814409 28514 raft_consensus.cc:2781] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:08.814738 28514 raft_consensus.cc:482] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:08.815076 28514 raft_consensus.cc:3037] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:08.815685 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b09a379fab4ad6aa1985c84fe6654a" candidate_uuid: "9c63787081744750872196ba0013abed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" is_pre_election: true
I20260810 22:04:08.815845 28502 raft_consensus.cc:482] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:08.816321 28352 raft_consensus.cc:2454] T 19b09a379fab4ad6aa1985c84fe6654a P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c63787081744750872196ba0013abed in term 0.
I20260810 22:04:08.816480 28502 raft_consensus.cc:504] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.817765 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "312e5bba4bc240d39b6d15652a40cc79" candidate_uuid: "9c63787081744750872196ba0013abed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" is_pre_election: true
I20260810 22:04:08.818460 28351 raft_consensus.cc:2454] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c63787081744750872196ba0013abed in term 0.
I20260810 22:04:08.820640 28072 leader_election.cc:302] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 050f5ddd2d7c44c8a923f9f876d4e186, 9c63787081744750872196ba0013abed; no voters:
W20260810 22:04:08.823868 28041 tablet.cc:2369] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:04:08.828439 28514 raft_consensus.cc:504] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.830857 28502 leader_election.cc:288] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487), 9c63787081744750872196ba0013abed (127.27.17.130:37287)
I20260810 22:04:08.831899 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b09a379fab4ad6aa1985c84fe6654a" candidate_uuid: "9c63787081744750872196ba0013abed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0fbda82fb84730a663338f3552d36a"
I20260810 22:04:08.832280 28158 raft_consensus.cc:2454] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b0fbda82fb84730a663338f3552d36a in term 1.
I20260810 22:04:08.832687 27981 raft_consensus.cc:3037] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:08.832775 28505 raft_consensus.cc:2741] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260810 22:04:08.836006 27888 leader_election.cc:302] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b0fbda82fb84730a663338f3552d36a, 9c63787081744750872196ba0013abed; no voters:
I20260810 22:04:08.836395 28479 leader_election.cc:288] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [CANDIDATE]: Term 1 election: Requested vote from peers 9c63787081744750872196ba0013abed (127.27.17.130:37287), 050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131:36895)
I20260810 22:04:08.836988 28476 raft_consensus.cc:2781] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:08.837374 28514 leader_election.cc:288] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed [CANDIDATE]: Term 1 election: Requested vote from peers 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487), 050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131:36895)
I20260810 22:04:08.837486 28476 raft_consensus.cc:686] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [term 1 LEADER]: Becoming Leader. State: Replica: 8b0fbda82fb84730a663338f3552d36a, State: Running, Role: LEADER
I20260810 22:04:08.838524 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b09a379fab4ad6aa1985c84fe6654a" candidate_uuid: "9c63787081744750872196ba0013abed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "050f5ddd2d7c44c8a923f9f876d4e186"
I20260810 22:04:08.838254 28476 consensus_queue.cc:237] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.839155 28351 raft_consensus.cc:3037] T 19b09a379fab4ad6aa1985c84fe6654a P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:08.840148 27981 raft_consensus.cc:2454] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c63787081744750872196ba0013abed in term 1.
I20260810 22:04:08.841598 28075 leader_election.cc:302] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b0fbda82fb84730a663338f3552d36a, 9c63787081744750872196ba0013abed; no voters:
I20260810 22:04:08.843042 28514 raft_consensus.cc:2781] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:08.844625 28514 raft_consensus.cc:686] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed [term 1 LEADER]: Becoming Leader. State: Replica: 9c63787081744750872196ba0013abed, State: Running, Role: LEADER
I20260810 22:04:08.847505 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "312e5bba4bc240d39b6d15652a40cc79" candidate_uuid: "8b0fbda82fb84730a663338f3552d36a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "050f5ddd2d7c44c8a923f9f876d4e186"
I20260810 22:04:08.848096 28352 raft_consensus.cc:3037] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:08.847621 28514 consensus_queue.cc:237] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.852748 28158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140448518535465eafa77555fa59db8c" candidate_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c63787081744750872196ba0013abed" is_pre_election: true
I20260810 22:04:08.853403 28158 raft_consensus.cc:2454] T 140448518535465eafa77555fa59db8c P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 050f5ddd2d7c44c8a923f9f876d4e186 in term 0.
I20260810 22:04:08.854799 28256 leader_election.cc:302] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 050f5ddd2d7c44c8a923f9f876d4e186, 9c63787081744750872196ba0013abed; no voters:
I20260810 22:04:08.855700 28502 raft_consensus.cc:2781] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:08.856043 28502 raft_consensus.cc:482] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:08.856400 28502 raft_consensus.cc:3037] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:08.856472 28351 raft_consensus.cc:2454] T 19b09a379fab4ad6aa1985c84fe6654a P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c63787081744750872196ba0013abed in term 1.
I20260810 22:04:08.864415 28352 raft_consensus.cc:2454] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b0fbda82fb84730a663338f3552d36a in term 1.
I20260810 22:04:08.866920 28502 raft_consensus.cc:504] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.866525 27779 catalog_manager.cc:5495] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b0fbda82fb84730a663338f3552d36a (127.27.17.129). New cstate: current_term: 1 leader_uuid: "8b0fbda82fb84730a663338f3552d36a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:08.867374 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140448518535465eafa77555fa59db8c" candidate_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0fbda82fb84730a663338f3552d36a" is_pre_election: true
I20260810 22:04:08.866396 27780 catalog_manager.cc:5495] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c63787081744750872196ba0013abed (127.27.17.130). New cstate: current_term: 1 leader_uuid: "9c63787081744750872196ba0013abed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:08.867982 27981 raft_consensus.cc:2454] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 050f5ddd2d7c44c8a923f9f876d4e186 in term 0.
I20260810 22:04:08.869586 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140448518535465eafa77555fa59db8c" candidate_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0fbda82fb84730a663338f3552d36a"
I20260810 22:04:08.870132 27981 raft_consensus.cc:3037] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:08.870489 28502 leader_election.cc:288] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 1 election: Requested vote from peers 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487), 9c63787081744750872196ba0013abed (127.27.17.130:37287)
I20260810 22:04:08.870723 28510 raft_consensus.cc:482] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:08.871161 28510 raft_consensus.cc:504] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.871760 28158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140448518535465eafa77555fa59db8c" candidate_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c63787081744750872196ba0013abed"
I20260810 22:04:08.872363 28158 raft_consensus.cc:3037] T 140448518535465eafa77555fa59db8c P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:08.873210 28510 leader_election.cc:288] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487), 9c63787081744750872196ba0013abed (127.27.17.130:37287)
I20260810 22:04:08.875168 27982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5bd5361adba43f09bdf5c51cda4bbcd" candidate_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0fbda82fb84730a663338f3552d36a" is_pre_election: true
I20260810 22:04:08.875965 27982 raft_consensus.cc:2454] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 050f5ddd2d7c44c8a923f9f876d4e186 in term 0.
I20260810 22:04:08.876358 27981 raft_consensus.cc:2454] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 050f5ddd2d7c44c8a923f9f876d4e186 in term 1.
I20260810 22:04:08.877761 28257 leader_election.cc:302] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 050f5ddd2d7c44c8a923f9f876d4e186, 8b0fbda82fb84730a663338f3552d36a; no voters:
I20260810 22:04:08.877780 28156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5bd5361adba43f09bdf5c51cda4bbcd" candidate_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c63787081744750872196ba0013abed" is_pre_election: true
I20260810 22:04:08.878508 28156 raft_consensus.cc:2454] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 050f5ddd2d7c44c8a923f9f876d4e186 in term 0.
I20260810 22:04:08.878711 28257 leader_election.cc:302] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 050f5ddd2d7c44c8a923f9f876d4e186, 8b0fbda82fb84730a663338f3552d36a; no voters:
I20260810 22:04:08.879717 28510 raft_consensus.cc:2781] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:08.880111 28510 raft_consensus.cc:482] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:08.880527 28510 raft_consensus.cc:3037] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:08.880359 28158 raft_consensus.cc:2454] T 140448518535465eafa77555fa59db8c P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 050f5ddd2d7c44c8a923f9f876d4e186 in term 1.
I20260810 22:04:08.885138 28502 raft_consensus.cc:2781] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:08.885599 28502 raft_consensus.cc:686] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 LEADER]: Becoming Leader. State: Replica: 050f5ddd2d7c44c8a923f9f876d4e186, State: Running, Role: LEADER
I20260810 22:04:08.886289 28510 raft_consensus.cc:504] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.886492 28502 consensus_queue.cc:237] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.888183 28510 leader_election.cc:288] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 1 election: Requested vote from peers 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487), 9c63787081744750872196ba0013abed (127.27.17.130:37287)
I20260810 22:04:08.888616 27981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5bd5361adba43f09bdf5c51cda4bbcd" candidate_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0fbda82fb84730a663338f3552d36a"
I20260810 22:04:08.888892 28158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5bd5361adba43f09bdf5c51cda4bbcd" candidate_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c63787081744750872196ba0013abed"
I20260810 22:04:08.889175 27981 raft_consensus.cc:3037] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:08.889441 28158 raft_consensus.cc:3037] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:08.895478 28158 raft_consensus.cc:2454] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 050f5ddd2d7c44c8a923f9f876d4e186 in term 1.
I20260810 22:04:08.896481 28256 leader_election.cc:302] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 050f5ddd2d7c44c8a923f9f876d4e186, 9c63787081744750872196ba0013abed; no voters:
I20260810 22:04:08.896581 27981 raft_consensus.cc:2454] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 050f5ddd2d7c44c8a923f9f876d4e186 in term 1.
I20260810 22:04:08.897246 28502 raft_consensus.cc:2781] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:08.897737 28502 raft_consensus.cc:686] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 LEADER]: Becoming Leader. State: Replica: 050f5ddd2d7c44c8a923f9f876d4e186, State: Running, Role: LEADER
I20260810 22:04:08.898378 28502 consensus_queue.cc:237] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:08.898483 27780 catalog_manager.cc:5495] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 reported cstate change: term changed from 0 to 1, leader changed from <none> to 050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131). New cstate: current_term: 1 leader_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
W20260810 22:04:08.903278 28423 tablet.cc:2369] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:04:08.911080 27780 catalog_manager.cc:5495] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 reported cstate change: term changed from 0 to 1, leader changed from <none> to 050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131). New cstate: current_term: 1 leader_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:09.187489 28476 consensus_queue.cc:1035] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 22:04:09.216540 28476 consensus_queue.cc:1035] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [LEADER]: Connected to new peer: Peer: permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 22:04:09.260686 28476 consensus_queue.cc:1035] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 22:04:09.279618 28502 consensus_queue.cc:1035] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 22:04:09.292289 28479 consensus_queue.cc:1035] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [LEADER]: Connected to new peer: Peer: permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 22:04:09.305593 28510 consensus_queue.cc:1035] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:04:09.330756 28514 consensus_queue.cc:1035] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 22:04:09.356660 28510 consensus_queue.cc:1035] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:04:09.380961 28510 consensus_queue.cc:1035] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 22:04:09.434484 28514 consensus_queue.cc:1035] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed [LEADER]: Connected to new peer: Peer: permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 22:04:10.268534 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 27858
I20260810 22:04:10.323261 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.129:41487
--local_ip_for_outbound_sockets=127.27.17.129
--tserver_master_addrs=127.27.17.190:34181
--webserver_port=42257
--webserver_interface=127.27.17.129
--env_inject_eio_globs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1/**
--env_inject_eio=1.0
--crash_on_eio=false
--builtin_ntp_servers=127.27.17.148:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20260810 22:04:10.381042 28257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.129:41487: connect: Connection refused (error 111)
W20260810 22:04:10.384094 28257 consensus_peers.cc:487] T 140448518535465eafa77555fa59db8c P 050f5ddd2d7c44c8a923f9f876d4e186 -> Peer 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): Couldn't send request to peer 8b0fbda82fb84730a663338f3552d36a. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:41487: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 22:04:10.529382 28075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.129:41487: connect: Connection refused (error 111)
W20260810 22:04:10.532586 28075 consensus_peers.cc:487] T 19b09a379fab4ad6aa1985c84fe6654a P 9c63787081744750872196ba0013abed -> Peer 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): Couldn't send request to peer 8b0fbda82fb84730a663338f3552d36a. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:41487: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 22:04:10.647212 28569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:10.647782 28569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:10.648418 28569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:10.680388 28569 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260810 22:04:10.680853 28569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:10.681524 28569 flags.cc:425] Enabled experimental flag: --crash_on_eio=false
W20260810 22:04:10.682092 28569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.129
I20260810 22:04:10.736423 28569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.129:41487
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.27.17.129
--webserver_port=42257
--tserver_master_addrs=127.27.17.190:34181
--env_inject_eio=1
--env_inject_eio_globs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1/**
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.129
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:10.737735 28569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260810 22:04:10.738631 28569 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260810 22:04:10.738770 28569 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260810 22:04:10.739791 28569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:10.758574 28580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:10.762943 28257 consensus_peers.cc:487] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 -> Peer 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): Couldn't send request to peer 8b0fbda82fb84730a663338f3552d36a. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:41487: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:11.331349 28566 raft_consensus.cc:482] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b0fbda82fb84730a663338f3552d36a)
I20260810 22:04:11.331926 28566 raft_consensus.cc:504] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:11.333501 28566 leader_election.cc:288] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487), 050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131:36895)
I20260810 22:04:11.334717 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c849b79d77074d9eb6cfe7699ac8e78e" candidate_uuid: "9c63787081744750872196ba0013abed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" is_pre_election: true
W20260810 22:04:11.337163 28075 leader_election.cc:334] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): Network error: Client connection negotiation failed: client connection to 127.27.17.129:41487: connect: Connection refused (error 111)
I20260810 22:04:11.337587 28075 leader_election.cc:302] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c63787081744750872196ba0013abed; no voters: 050f5ddd2d7c44c8a923f9f876d4e186, 8b0fbda82fb84730a663338f3552d36a
I20260810 22:04:11.338184 28566 raft_consensus.cc:2726] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:04:11.364950 28566 raft_consensus.cc:482] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b0fbda82fb84730a663338f3552d36a)
I20260810 22:04:11.365453 28566 raft_consensus.cc:504] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:11.367071 28566 leader_election.cc:288] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487), 050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131:36895)
I20260810 22:04:11.368638 28352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "312e5bba4bc240d39b6d15652a40cc79" candidate_uuid: "9c63787081744750872196ba0013abed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" is_pre_election: true
W20260810 22:04:11.370566 28075 leader_election.cc:334] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): Network error: Client connection negotiation failed: client connection to 127.27.17.129:41487: connect: Connection refused (error 111)
I20260810 22:04:11.370934 28075 leader_election.cc:302] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c63787081744750872196ba0013abed; no voters: 050f5ddd2d7c44c8a923f9f876d4e186, 8b0fbda82fb84730a663338f3552d36a
I20260810 22:04:11.371505 28566 raft_consensus.cc:2726] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:04:11.386462 28510 raft_consensus.cc:482] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b0fbda82fb84730a663338f3552d36a)
I20260810 22:04:11.386945 28510 raft_consensus.cc:504] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:11.388760 28510 leader_election.cc:288] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487), 9c63787081744750872196ba0013abed (127.27.17.130:37287)
I20260810 22:04:11.390595 28158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "312e5bba4bc240d39b6d15652a40cc79" candidate_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c63787081744750872196ba0013abed" is_pre_election: true
I20260810 22:04:11.391834 28158 raft_consensus.cc:2454] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 050f5ddd2d7c44c8a923f9f876d4e186 in term 1.
W20260810 22:04:11.392624 28257 leader_election.cc:334] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): Network error: Client connection negotiation failed: client connection to 127.27.17.129:41487: connect: Connection refused (error 111)
I20260810 22:04:11.393442 28256 leader_election.cc:302] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 050f5ddd2d7c44c8a923f9f876d4e186, 9c63787081744750872196ba0013abed; no voters: 8b0fbda82fb84730a663338f3552d36a
I20260810 22:04:11.394066 28510 raft_consensus.cc:2781] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 22:04:11.394392 28510 raft_consensus.cc:482] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8b0fbda82fb84730a663338f3552d36a)
I20260810 22:04:11.394677 28510 raft_consensus.cc:3037] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:04:11.399605 28510 raft_consensus.cc:504] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:11.402035 28510 leader_election.cc:288] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 2 election: Requested vote from peers 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487), 9c63787081744750872196ba0013abed (127.27.17.130:37287)
I20260810 22:04:11.403671 28158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "312e5bba4bc240d39b6d15652a40cc79" candidate_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c63787081744750872196ba0013abed"
I20260810 22:04:11.404507 28158 raft_consensus.cc:3037] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Advancing to term 2
W20260810 22:04:11.406390 28257 leader_election.cc:334] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): Network error: Client connection negotiation failed: client connection to 127.27.17.129:41487: connect: Connection refused (error 111)
I20260810 22:04:11.414228 28158 raft_consensus.cc:2454] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 050f5ddd2d7c44c8a923f9f876d4e186 in term 2.
I20260810 22:04:11.418212 28256 leader_election.cc:302] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 050f5ddd2d7c44c8a923f9f876d4e186, 9c63787081744750872196ba0013abed; no voters: 8b0fbda82fb84730a663338f3552d36a
I20260810 22:04:11.418923 28590 raft_consensus.cc:2781] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:04:11.419389 28590 raft_consensus.cc:686] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [term 2 LEADER]: Becoming Leader. State: Replica: 050f5ddd2d7c44c8a923f9f876d4e186, State: Running, Role: LEADER
I20260810 22:04:11.421499 28590 consensus_queue.cc:237] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:11.429952 27779 catalog_manager.cc:5495] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 reported cstate change: term changed from 1 to 2, leader changed from 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) to 050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131). New cstate: current_term: 2 leader_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:11.698586 28590 raft_consensus.cc:482] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b0fbda82fb84730a663338f3552d36a)
I20260810 22:04:11.699054 28590 raft_consensus.cc:504] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:11.702258 28158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c849b79d77074d9eb6cfe7699ac8e78e" candidate_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c63787081744750872196ba0013abed" is_pre_election: true
I20260810 22:04:11.702907 28158 raft_consensus.cc:2454] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 050f5ddd2d7c44c8a923f9f876d4e186 in term 1.
I20260810 22:04:11.704191 28256 leader_election.cc:302] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 050f5ddd2d7c44c8a923f9f876d4e186, 9c63787081744750872196ba0013abed; no voters:
I20260810 22:04:11.704993 28510 raft_consensus.cc:2781] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260810 22:04:11.705195 28257 leader_election.cc:334] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): Network error: Client connection negotiation failed: client connection to 127.27.17.129:41487: connect: Connection refused (error 111)
I20260810 22:04:11.705322 28510 raft_consensus.cc:482] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8b0fbda82fb84730a663338f3552d36a)
I20260810 22:04:11.705685 28510 raft_consensus.cc:3037] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:04:11.707669 28590 leader_election.cc:288] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487), 9c63787081744750872196ba0013abed (127.27.17.130:37287)
I20260810 22:04:11.710351 28510 raft_consensus.cc:504] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:11.713027 28158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c849b79d77074d9eb6cfe7699ac8e78e" candidate_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9c63787081744750872196ba0013abed"
I20260810 22:04:11.713636 28158 raft_consensus.cc:3037] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Advancing to term 2
W20260810 22:04:11.715781 28257 leader_election.cc:334] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): Network error: Client connection negotiation failed: client connection to 127.27.17.129:41487: connect: Connection refused (error 111)
I20260810 22:04:11.716504 28510 leader_election.cc:288] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 2 election: Requested vote from peers 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487), 9c63787081744750872196ba0013abed (127.27.17.130:37287)
I20260810 22:04:11.719566 28158 raft_consensus.cc:2454] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 050f5ddd2d7c44c8a923f9f876d4e186 in term 2.
I20260810 22:04:11.721092 28256 leader_election.cc:302] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 050f5ddd2d7c44c8a923f9f876d4e186, 9c63787081744750872196ba0013abed; no voters: 8b0fbda82fb84730a663338f3552d36a
I20260810 22:04:11.721937 28510 raft_consensus.cc:2781] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:04:11.724167 28510 raft_consensus.cc:686] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [term 2 LEADER]: Becoming Leader. State: Replica: 050f5ddd2d7c44c8a923f9f876d4e186, State: Running, Role: LEADER
I20260810 22:04:11.724992 28510 consensus_queue.cc:237] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:11.732141 27779 catalog_manager.cc:5495] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 reported cstate change: term changed from 1 to 2, leader changed from 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) to 050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131). New cstate: current_term: 2 leader_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:11.806614 28158 raft_consensus.cc:1261] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 2 FOLLOWER]: Refusing update from remote peer 050f5ddd2d7c44c8a923f9f876d4e186: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 22:04:11.807858 28510 consensus_queue.cc:1035] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260810 22:04:11.828578 28257 consensus_peers.cc:487] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 -> Peer 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): Couldn't send request to peer 8b0fbda82fb84730a663338f3552d36a. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:41487: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 22:04:10.759995 28584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:10.759008 28581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:11.999418 28582 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 22:04:11.999477 28569 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 22:04:12.004144 28569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:12.006590 28569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:12.007956 28569 hybrid_clock.cc:648] HybridClock initialized: now 1786399452007927 us; error 53 us; skew 500 ppm
I20260810 22:04:12.008795 28569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:12.016065 28569 webserver.cc:460] Webserver started at http://127.27.17.129:42257/ using document root <none> and password file <none>
I20260810 22:04:12.017843 28569 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:12.018116 28569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
E20260810 22:04:12.020691 28569 env_posix.cc:432] I/O error, context: INJECTED FAILURE
E20260810 22:04:12.025916 28569 env_posix.cc:432] I/O error, context: INJECTED FAILURE
W20260810 22:04:12.038336 28569 data_dirs.cc:318] instance /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1/data/block_manager_instance has unknown UUID 5e314b01e35644e888f8beb5518faf90
I20260810 22:04:12.039536 28569 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.011s sys 0.005s
I20260810 22:04:12.041997 28569 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:12.042255 28569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/wal
uuid: "8b0fbda82fb84730a663338f3552d36a"
format_stamp: "Formatted at 2026-08-10 22:04:04 on dist-test-slave-bw88"
I20260810 22:04:12.045329 28569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/wal
4 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260810 22:04:12.089417 28569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:12.091281 28569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:12.091944 28569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:12.095475 28569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:04:12.103440 28624 ts_tablet_manager.cc:542] Loading tablet metadata (0/5 complete)
I20260810 22:04:12.123483 28569 ts_tablet_manager.cc:579] Loaded tablet metadata (5 total tablets, 5 live tablets)
I20260810 22:04:12.123756 28569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.023s user 0.006s sys 0.009s
I20260810 22:04:12.124135 28569 ts_tablet_manager.cc:594] Registering tablets (0/5 complete)
I20260810 22:04:12.129951 28624 tablet_bootstrap.cc:492] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: Bootstrap starting.
E20260810 22:04:12.130664 28624 ts_tablet_manager.cc:1419] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory
I20260810 22:04:12.130939 28624 ts_tablet_manager.cc:1397] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:04:12.131251 28624 tablet_replica.cc:330] stopping tablet replica
I20260810 22:04:12.133725 28628 tablet_bootstrap.cc:492] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a: Bootstrap starting.
I20260810 22:04:12.131534 28624 raft_consensus.cc:2229] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:04:12.134377 28624 raft_consensus.cc:2258] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:04:12.139559 28624 tablet_bootstrap.cc:492] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: Bootstrap starting.
E20260810 22:04:12.140076 28624 ts_tablet_manager.cc:1419] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory
I20260810 22:04:12.140329 28624 ts_tablet_manager.cc:1397] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:04:12.140565 28624 tablet_replica.cc:330] stopping tablet replica
I20260810 22:04:12.140823 28624 raft_consensus.cc:2229] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:04:12.141149 28624 raft_consensus.cc:2258] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 22:04:12.146942 28257 consensus_peers.cc:487] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 -> Peer 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): Couldn't send request to peer 8b0fbda82fb84730a663338f3552d36a. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:41487: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:12.147933 28624 tablet_bootstrap.cc:492] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a: Bootstrap starting.
I20260810 22:04:12.152118 28569 ts_tablet_manager.cc:610] Registered 5 tablets
I20260810 22:04:12.152539 28569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.028s user 0.025s sys 0.004s
I20260810 22:04:12.154407 28629 tablet_bootstrap.cc:492] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: Bootstrap starting.
E20260810 22:04:12.155236 28629 ts_tablet_manager.cc:1419] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory
I20260810 22:04:12.155671 28629 ts_tablet_manager.cc:1397] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 22:04:12.163424 28629 tablet_replica.cc:330] stopping tablet replica
I20260810 22:04:12.163811 28629 raft_consensus.cc:2229] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:04:12.164295 28629 raft_consensus.cc:2258] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:04:12.219933 28624 log.cc:824] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:12.237418 28624 tablet_bootstrap.cc:492] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:04:12.239740 28624 tablet_bootstrap.cc:492] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a: Bootstrap complete.
I20260810 22:04:12.242086 28624 ts_tablet_manager.cc:1397] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a: Time spent bootstrapping tablet: real 0.094s user 0.051s sys 0.031s
I20260810 22:04:12.247426 28628 tablet_bootstrap.cc:492] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:04:12.249377 28628 tablet_bootstrap.cc:492] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a: Bootstrap complete.
I20260810 22:04:12.251713 28628 ts_tablet_manager.cc:1397] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a: Time spent bootstrapping tablet: real 0.118s user 0.053s sys 0.036s
I20260810 22:04:12.278290 28628 raft_consensus.cc:348] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:12.279517 28628 raft_consensus.cc:729] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0fbda82fb84730a663338f3552d36a, State: Initialized, Role: FOLLOWER
I20260810 22:04:12.280759 28628 consensus_queue.cc:260] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:12.283214 28624 raft_consensus.cc:348] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:12.284171 28624 raft_consensus.cc:729] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0fbda82fb84730a663338f3552d36a, State: Initialized, Role: FOLLOWER
I20260810 22:04:12.285235 28624 consensus_queue.cc:260] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:12.298411 28158 raft_consensus.cc:1261] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 2 FOLLOWER]: Refusing update from remote peer 050f5ddd2d7c44c8a923f9f876d4e186: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 22:04:12.299268 28624 ts_tablet_manager.cc:1428] T 140448518535465eafa77555fa59db8c P 8b0fbda82fb84730a663338f3552d36a: Time spent starting tablet: real 0.057s user 0.035s sys 0.009s
I20260810 22:04:12.300957 28628 ts_tablet_manager.cc:1428] T 19b09a379fab4ad6aa1985c84fe6654a P 8b0fbda82fb84730a663338f3552d36a: Time spent starting tablet: real 0.049s user 0.011s sys 0.011s
I20260810 22:04:12.302016 28590 consensus_queue.cc:1035] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 22:04:12.435874 28569 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.129:41487
I20260810 22:04:12.436156 28788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.129:41487 every 8 connection(s)
I20260810 22:04:12.439481 28569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
I20260810 22:04:12.445830 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 28569
I20260810 22:04:12.461474 28789 heartbeater.cc:347] Connected to a master server at 127.27.17.190:34181
I20260810 22:04:12.461927 28789 heartbeater.cc:464] Registering TS with master...
I20260810 22:04:12.462869 28789 heartbeater.cc:511] Master 127.27.17.190:34181 requested a full tablet report, sending...
I20260810 22:04:12.467283 27779 ts_manager.cc:194] Re-registered known tserver with Master: 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487)
W20260810 22:04:12.469707 27779 catalog_manager.cc:5408] Tablet 312e5bba4bc240d39b6d15652a40cc79 (table test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]) has failed on TS 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): IO error: some tablet data is in a failed directory
W20260810 22:04:12.470268 27779 catalog_manager.cc:5408] Tablet c849b79d77074d9eb6cfe7699ac8e78e (table test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]) has failed on TS 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): IO error: some tablet data is in a failed directory
W20260810 22:04:12.470726 27779 catalog_manager.cc:5408] Tablet b5bd5361adba43f09bdf5c51cda4bbcd (table test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]) has failed on TS 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): IO error: some tablet data is in a failed directory
I20260810 22:04:12.473310 27779 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.129:54427
I20260810 22:04:12.542560 28510 raft_consensus.cc:998] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186: Attempting to remove follower 8b0fbda82fb84730a663338f3552d36a from the Raft config. Reason: The tablet replica hosted on peer 8b0fbda82fb84730a663338f3552d36a has failed
I20260810 22:04:12.544471 28510 consensus_queue.cc:237] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:12.549288 28158 raft_consensus.cc:1261] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Refusing update from remote peer 050f5ddd2d7c44c8a923f9f876d4e186: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:04:12.550751 28590 consensus_queue.cc:1035] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 22:04:12.557670 28510 raft_consensus.cc:2932] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:12.559393 28158 raft_consensus.cc:2932] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:12.573807 27779 catalog_manager.cc:5495] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed reported cstate change: config changed from index -1 to 2, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) evicted. New cstate: current_term: 1 leader_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:12.576390 28510 raft_consensus.cc:998] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186: Attempting to remove follower 8b0fbda82fb84730a663338f3552d36a from the Raft config. Reason: The tablet replica hosted on peer 8b0fbda82fb84730a663338f3552d36a has failed
I20260810 22:04:12.578503 28510 consensus_queue.cc:237] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:12.588979 28158 raft_consensus.cc:1261] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 2 FOLLOWER]: Refusing update from remote peer 050f5ddd2d7c44c8a923f9f876d4e186: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260810 22:04:12.590765 28807 consensus_queue.cc:1035] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 22:04:12.599005 28808 raft_consensus.cc:2932] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:12.600678 28352 consensus_queue.cc:237] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } }
W20260810 22:04:12.608502 28257 consensus_peers.cc:487] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 -> Peer 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): Couldn't send request to peer 8b0fbda82fb84730a663338f3552d36a. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: some tablet data is in a failed directory. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:12.609686 28158 raft_consensus.cc:2932] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:12.610412 28156 raft_consensus.cc:1261] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Refusing update from remote peer 050f5ddd2d7c44c8a923f9f876d4e186: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 22:04:12.612263 27779 catalog_manager.cc:5495] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 reported cstate change: config changed from index -1 to 3, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) evicted. New cstate: current_term: 2 leader_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:12.615952 28808 consensus_queue.cc:1035] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 22:04:12.623042 28807 raft_consensus.cc:2932] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } }
I20260810 22:04:12.629227 28352 consensus_queue.cc:237] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } }
I20260810 22:04:12.634807 27760 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:VOTER RPC for tablet b5bd5361adba43f09bdf5c51cda4bbcd with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20260810 22:04:12.637682 28257 consensus_peers.cc:487] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 -> Peer 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): Couldn't send request to peer 8b0fbda82fb84730a663338f3552d36a. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: some tablet data is in a failed directory. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:12.639732 28156 raft_consensus.cc:1261] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 2 FOLLOWER]: Refusing update from remote peer 050f5ddd2d7c44c8a923f9f876d4e186: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260810 22:04:12.641083 28695 tablet_service.cc:1514] Processing DeleteTablet for tablet c849b79d77074d9eb6cfe7699ac8e78e with delete_type TABLET_DATA_TOMBSTONED (TS 8b0fbda82fb84730a663338f3552d36a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41270
I20260810 22:04:12.641271 28806 consensus_queue.cc:1035] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 22:04:12.641809 28696 tablet_service.cc:1514] Processing DeleteTablet for tablet b5bd5361adba43f09bdf5c51cda4bbcd with delete_type TABLET_DATA_TOMBSTONED (TS 8b0fbda82fb84730a663338f3552d36a not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:41270
I20260810 22:04:12.641388 28158 raft_consensus.cc:2932] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } }
I20260810 22:04:12.645545 27779 catalog_manager.cc:5495] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 reported cstate change: config changed from index 2 to 3, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) added. New cstate: current_term: 1 leader_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } }
I20260810 22:04:12.655561 28808 raft_consensus.cc:2932] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } }
I20260810 22:04:12.661267 28156 raft_consensus.cc:2932] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } }
I20260810 22:04:12.663388 28816 ts_tablet_manager.cc:1905] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 22:04:12.666288 27760 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:VOTER RPC for tablet c849b79d77074d9eb6cfe7699ac8e78e with cas_config_opid_index 3: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260810 22:04:12.667237 28815 ts_tablet_manager.cc:1905] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 22:04:12.672019 27778 catalog_manager.cc:5495] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 reported cstate change: config changed from index 3 to 4, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) added. New cstate: current_term: 2 leader_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } }
I20260810 22:04:12.684677 28789 heartbeater.cc:503] Master 127.27.17.190:34181 was elected leader, sending a full tablet report...
I20260810 22:04:12.685166 28816 ts_tablet_manager.cc:1918] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260810 22:04:12.685617 28816 log.cc:1196] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: Deleting WAL directory at /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/wal/wals/c849b79d77074d9eb6cfe7699ac8e78e
I20260810 22:04:12.688565 27764 catalog_manager.cc:4841] TS 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): tablet c849b79d77074d9eb6cfe7699ac8e78e (table test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]) successfully deleted
W20260810 22:04:12.691458 27778 catalog_manager.cc:5408] Tablet 312e5bba4bc240d39b6d15652a40cc79 (table test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]) has failed on TS 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): IO error: some tablet data is in a failed directory
I20260810 22:04:12.691826 28815 ts_tablet_manager.cc:1918] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260810 22:04:12.692267 28815 log.cc:1196] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: Deleting WAL directory at /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/wal/wals/b5bd5361adba43f09bdf5c51cda4bbcd
W20260810 22:04:12.692356 27778 catalog_manager.cc:5408] Tablet c849b79d77074d9eb6cfe7699ac8e78e (table test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]) has failed on TS 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): IO error: some tablet data is in a failed directory
W20260810 22:04:12.693221 27778 catalog_manager.cc:5408] Tablet b5bd5361adba43f09bdf5c51cda4bbcd (table test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]) has failed on TS 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): IO error: some tablet data is in a failed directory
I20260810 22:04:12.695195 27764 catalog_manager.cc:4841] TS 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): tablet b5bd5361adba43f09bdf5c51cda4bbcd (table test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]) successfully deleted
I20260810 22:04:12.892390 28808 raft_consensus.cc:998] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186: Attempting to remove follower 8b0fbda82fb84730a663338f3552d36a from the Raft config. Reason: The tablet replica hosted on peer 8b0fbda82fb84730a663338f3552d36a has failed
I20260810 22:04:12.895108 28808 consensus_queue.cc:237] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:12.905498 28156 raft_consensus.cc:1261] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 2 FOLLOWER]: Refusing update from remote peer 050f5ddd2d7c44c8a923f9f876d4e186: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260810 22:04:12.907495 28695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 22:04:12.910604 28808 consensus_queue.cc:1035] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 22:04:12.913077 28317 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20260810 22:04:12.915863 28132 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 22:04:12.921785 28807 raft_consensus.cc:2932] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:12.927058 28158 raft_consensus.cc:2932] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:12.950994 27779 catalog_manager.cc:5495] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed reported cstate change: config changed from index -1 to 3, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) evicted. New cstate: current_term: 2 leader_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:12.971081 28695 tablet_service.cc:1514] Processing DeleteTablet for tablet 312e5bba4bc240d39b6d15652a40cc79 with delete_type TABLET_DATA_TOMBSTONED (TS 8b0fbda82fb84730a663338f3552d36a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41270
I20260810 22:04:12.972623 28815 ts_tablet_manager.cc:1905] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 22:04:12.987922 28352 consensus_queue.cc:237] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } }
W20260810 22:04:12.996521 28257 consensus_peers.cc:487] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 -> Peer 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): Couldn't send request to peer 8b0fbda82fb84730a663338f3552d36a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:12.999739 28158 raft_consensus.cc:1261] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 2 FOLLOWER]: Refusing update from remote peer 050f5ddd2d7c44c8a923f9f876d4e186: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260810 22:04:13.001312 28807 consensus_queue.cc:1035] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 22:04:13.009625 28808 raft_consensus.cc:2932] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } }
I20260810 22:04:13.012455 28156 raft_consensus.cc:2932] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } }
I20260810 22:04:13.021993 28815 ts_tablet_manager.cc:1918] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260810 22:04:13.022730 28815 log.cc:1196] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: Deleting WAL directory at /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/wal/wals/312e5bba4bc240d39b6d15652a40cc79
I20260810 22:04:13.025606 27764 catalog_manager.cc:4841] TS 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): tablet 312e5bba4bc240d39b6d15652a40cc79 (table test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]) successfully deleted
I20260810 22:04:13.038578 27779 catalog_manager.cc:5495] T 312e5bba4bc240d39b6d15652a40cc79 P 9c63787081744750872196ba0013abed reported cstate change: config changed from index 3 to 4, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) added. New cstate: current_term: 2 leader_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } }
I20260810 22:04:13.040537 27760 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:VOTER RPC for tablet 312e5bba4bc240d39b6d15652a40cc79 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260810 22:04:13.074029 28807 raft_consensus.cc:998] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186: Attempting to remove follower 8b0fbda82fb84730a663338f3552d36a from the Raft config. Reason: The tablet replica hosted on peer 8b0fbda82fb84730a663338f3552d36a has failed
I20260810 22:04:13.076045 28807 consensus_queue.cc:237] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:13.084000 28156 raft_consensus.cc:1261] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Refusing update from remote peer 050f5ddd2d7c44c8a923f9f876d4e186: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 22:04:13.085721 28807 consensus_queue.cc:1035] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 22:04:13.092809 28806 raft_consensus.cc:2932] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:13.102576 28156 raft_consensus.cc:2932] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:13.116621 27779 catalog_manager.cc:5495] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 reported cstate change: config changed from index 3 to 4, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) evicted. New cstate: current_term: 1 leader_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:13.129890 28808 raft_consensus.cc:998] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186: Attempting to remove follower 8b0fbda82fb84730a663338f3552d36a from the Raft config. Reason: The tablet replica hosted on peer 8b0fbda82fb84730a663338f3552d36a has failed
I20260810 22:04:13.131508 28808 consensus_queue.cc:237] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } }
I20260810 22:04:13.134582 28695 tablet_service.cc:1514] Processing DeleteTablet for tablet b5bd5361adba43f09bdf5c51cda4bbcd with delete_type TABLET_DATA_TOMBSTONED (TS 8b0fbda82fb84730a663338f3552d36a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41270
I20260810 22:04:13.136857 27764 catalog_manager.cc:4841] TS 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): tablet b5bd5361adba43f09bdf5c51cda4bbcd (table test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]) successfully deleted
I20260810 22:04:13.137480 28158 raft_consensus.cc:1261] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 2 FOLLOWER]: Refusing update from remote peer 050f5ddd2d7c44c8a923f9f876d4e186: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 22:04:13.139158 28808 consensus_queue.cc:1035] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 22:04:13.144171 28352 consensus_queue.cc:237] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } }
I20260810 22:04:13.146344 28808 raft_consensus.cc:2932] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:13.150199 28158 raft_consensus.cc:2932] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
W20260810 22:04:13.156330 28257 consensus_peers.cc:487] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 -> Peer 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): Couldn't send request to peer 8b0fbda82fb84730a663338f3552d36a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:13.163087 28155 raft_consensus.cc:1261] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Refusing update from remote peer 050f5ddd2d7c44c8a923f9f876d4e186: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 22:04:13.166005 28806 consensus_queue.cc:1035] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 22:04:13.174757 28807 raft_consensus.cc:2932] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } }
I20260810 22:04:13.189283 28155 raft_consensus.cc:2932] T b5bd5361adba43f09bdf5c51cda4bbcd P 9c63787081744750872196ba0013abed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } }
I20260810 22:04:13.192198 27779 catalog_manager.cc:5495] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed reported cstate change: config changed from index 4 to 5, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) evicted. New cstate: current_term: 2 leader_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } }
I20260810 22:04:13.213444 27760 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:VOTER RPC for tablet b5bd5361adba43f09bdf5c51cda4bbcd with cas_config_opid_index 4: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260810 22:04:13.225106 28695 tablet_service.cc:1514] Processing DeleteTablet for tablet c849b79d77074d9eb6cfe7699ac8e78e with delete_type TABLET_DATA_TOMBSTONED (TS 8b0fbda82fb84730a663338f3552d36a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41270
I20260810 22:04:13.227039 27764 catalog_manager.cc:4841] TS 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): tablet c849b79d77074d9eb6cfe7699ac8e78e (table test-workload [id=ef81c2e0989c45aa99f12390df2a43c5]) successfully deleted
I20260810 22:04:13.228192 28352 consensus_queue.cc:237] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } }
I20260810 22:04:13.231415 27779 catalog_manager.cc:5495] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186 reported cstate change: config changed from index 4 to 5, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) added. New cstate: current_term: 1 leader_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } }
W20260810 22:04:13.241791 28257 consensus_peers.cc:487] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 -> Peer 8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): Couldn't send request to peer 8b0fbda82fb84730a663338f3552d36a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:13.244908 28156 raft_consensus.cc:1261] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 2 FOLLOWER]: Refusing update from remote peer 050f5ddd2d7c44c8a923f9f876d4e186: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260810 22:04:13.248422 28808 consensus_queue.cc:1035] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 22:04:13.256567 28806 raft_consensus.cc:2932] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } }
I20260810 22:04:13.265679 27760 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:VOTER RPC for tablet c849b79d77074d9eb6cfe7699ac8e78e with cas_config_opid_index 5: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260810 22:04:13.270385 28156 raft_consensus.cc:2932] T c849b79d77074d9eb6cfe7699ac8e78e P 9c63787081744750872196ba0013abed [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } }
I20260810 22:04:13.275807 27779 catalog_manager.cc:5495] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186 reported cstate change: config changed from index 5 to 6, VOTER 8b0fbda82fb84730a663338f3552d36a (127.27.17.129) added. New cstate: current_term: 2 leader_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } } }
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
9a1244ce7efc411c91289f1ecd847644 | 127.27.17.190:34181 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:33389 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
050f5ddd2d7c44c8a923f9f876d4e186 | 127.27.17.131:36895 | HEALTHY | <none> | 4 | 0
8b0fbda82fb84730a663338f3552d36a | 127.27.17.129:41487 | HEALTHY | <none> | 0 | 0
9c63787081744750872196ba0013abed | 127.27.17.130:37287 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.27.17.129:41487
env_inject_eio | 1 | hidden | 127.27.17.129:41487
env_inject_eio_globs | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1/** | hidden | 127.27.17.129:41487
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.27.17.129 | experimental | 127.27.17.129:41487
local_ip_for_outbound_sockets | 127.27.17.130 | experimental | 127.27.17.130:37287
local_ip_for_outbound_sockets | 127.27.17.131 | experimental | 127.27.17.131:36895
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.27.17.129:41487
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.27.17.130:37287
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.27.17.131:36895
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:33389 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 312e5bba4bc240d39b6d15652a40cc79 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: some tablet data is in a failed directory
9c63787081744750872196ba0013abed (127.27.17.130:37287): RUNNING
050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131:36895): RUNNING [LEADER]
All reported replicas are:
A = 8b0fbda82fb84730a663338f3552d36a
B = 9c63787081744750872196ba0013abed
C = 050f5ddd2d7c44c8a923f9f876d4e186
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | B C* | 2 | 3 | No
C | B C* | 2 | 3 | No
Tablet c849b79d77074d9eb6cfe7699ac8e78e of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
9c63787081744750872196ba0013abed (127.27.17.130:37287): RUNNING
050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131:36895): RUNNING [LEADER]
8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 9c63787081744750872196ba0013abed
B = 050f5ddd2d7c44c8a923f9f876d4e186
C = 8b0fbda82fb84730a663338f3552d36a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 4 | Yes
B | A B* C | 2 | 4 | Yes
C | [config not available] | | |
Tablet b5bd5361adba43f09bdf5c51cda4bbcd of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
9c63787081744750872196ba0013abed (127.27.17.130:37287): RUNNING
050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131:36895): RUNNING [LEADER]
8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 9c63787081744750872196ba0013abed
B = 050f5ddd2d7c44c8a923f9f876d4e186
C = 8b0fbda82fb84730a663338f3552d36a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 3 | Yes
B | A B* C | 1 | 3 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 5 | 2 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 5
First Quartile | 5
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 22:04:13.378000 27718 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 22:04:13.405004 28848 ts_tablet_manager.cc:927] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: Initiating tablet copy from peer 050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131:36895)
I20260810 22:04:13.406369 28848 tablet_copy_client.cc:279] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid
I20260810 22:04:13.406895 28848 tablet_copy_client.cc:316] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.17.131:36895
I20260810 22:04:13.417079 28384 tablet_copy_service.cc:140] P 050f5ddd2d7c44c8a923f9f876d4e186: Received BeginTabletCopySession request for tablet 312e5bba4bc240d39b6d15652a40cc79 from peer 8b0fbda82fb84730a663338f3552d36a ({username='slave'} at 127.27.17.129:41907)
I20260810 22:04:13.417668 28384 tablet_copy_service.cc:161] P 050f5ddd2d7c44c8a923f9f876d4e186: Beginning new tablet copy session on tablet 312e5bba4bc240d39b6d15652a40cc79 from peer 8b0fbda82fb84730a663338f3552d36a at {username='slave'} at 127.27.17.129:41907: session id = 8b0fbda82fb84730a663338f3552d36a-312e5bba4bc240d39b6d15652a40cc79
I20260810 22:04:13.423077 28384 tablet_copy_source_session.cc:205] T 312e5bba4bc240d39b6d15652a40cc79 P 050f5ddd2d7c44c8a923f9f876d4e186: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:04:13.426128 28848 ts_tablet_manager.cc:1905] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260810 22:04:13.436676 28848 ts_tablet_manager.cc:1918] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown
I20260810 22:04:13.443037 28848 tablet_copy_client.cc:791] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: tablet copy: Starting download of 0 data blocks...
I20260810 22:04:13.443720 28848 tablet_copy_client.cc:655] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: tablet copy: Starting download of 1 WAL segments...
I20260810 22:04:13.448410 28848 tablet_copy_client.cc:531] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:04:13.454804 28848 tablet_bootstrap.cc:492] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: Bootstrap starting.
I20260810 22:04:13.480326 28848 tablet_bootstrap.cc:492] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:04:13.481503 28848 tablet_bootstrap.cc:492] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: Bootstrap complete.
I20260810 22:04:13.482178 28848 ts_tablet_manager.cc:1397] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.009s
I20260810 22:04:13.484859 28848 raft_consensus.cc:348] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } }
I20260810 22:04:13.485495 28848 raft_consensus.cc:729] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0fbda82fb84730a663338f3552d36a, State: Initialized, Role: FOLLOWER
I20260810 22:04:13.486173 28848 consensus_queue.cc:260] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } }
I20260810 22:04:13.490332 28848 ts_tablet_manager.cc:1428] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260810 22:04:13.492292 28384 tablet_copy_service.cc:342] P 050f5ddd2d7c44c8a923f9f876d4e186: Request end of tablet copy session 8b0fbda82fb84730a663338f3552d36a-312e5bba4bc240d39b6d15652a40cc79 received from {username='slave'} at 127.27.17.129:41907
I20260810 22:04:13.492647 28384 tablet_copy_service.cc:433] P 050f5ddd2d7c44c8a923f9f876d4e186: ending tablet copy session 8b0fbda82fb84730a663338f3552d36a-312e5bba4bc240d39b6d15652a40cc79 on tablet 312e5bba4bc240d39b6d15652a40cc79 with peer 8b0fbda82fb84730a663338f3552d36a
E20260810 22:04:13.499603 28785 env_posix.cc:432] I/O error, context: INJECTED FAILURE
I20260810 22:04:13.700124 28848 ts_tablet_manager.cc:927] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: Initiating tablet copy from peer 050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131:36895)
I20260810 22:04:13.705785 28874 ts_tablet_manager.cc:927] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: Initiating tablet copy from peer 050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131:36895)
I20260810 22:04:13.720403 28848 tablet_copy_client.cc:279] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid
I20260810 22:04:13.721071 28848 tablet_copy_client.cc:316] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.17.131:36895
I20260810 22:04:13.722885 28384 tablet_copy_service.cc:140] P 050f5ddd2d7c44c8a923f9f876d4e186: Received BeginTabletCopySession request for tablet c849b79d77074d9eb6cfe7699ac8e78e from peer 8b0fbda82fb84730a663338f3552d36a ({username='slave'} at 127.27.17.129:41907)
I20260810 22:04:13.723538 28384 tablet_copy_service.cc:161] P 050f5ddd2d7c44c8a923f9f876d4e186: Beginning new tablet copy session on tablet c849b79d77074d9eb6cfe7699ac8e78e from peer 8b0fbda82fb84730a663338f3552d36a at {username='slave'} at 127.27.17.129:41907: session id = 8b0fbda82fb84730a663338f3552d36a-c849b79d77074d9eb6cfe7699ac8e78e
I20260810 22:04:13.735361 28384 tablet_copy_source_session.cc:205] T c849b79d77074d9eb6cfe7699ac8e78e P 050f5ddd2d7c44c8a923f9f876d4e186: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:04:13.737198 28874 tablet_copy_client.cc:279] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid
I20260810 22:04:13.737967 28874 tablet_copy_client.cc:316] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.17.131:36895
I20260810 22:04:13.738373 28848 ts_tablet_manager.cc:1905] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260810 22:04:13.739812 28384 tablet_copy_service.cc:140] P 050f5ddd2d7c44c8a923f9f876d4e186: Received BeginTabletCopySession request for tablet b5bd5361adba43f09bdf5c51cda4bbcd from peer 8b0fbda82fb84730a663338f3552d36a ({username='slave'} at 127.27.17.129:41907)
I20260810 22:04:13.740399 28384 tablet_copy_service.cc:161] P 050f5ddd2d7c44c8a923f9f876d4e186: Beginning new tablet copy session on tablet b5bd5361adba43f09bdf5c51cda4bbcd from peer 8b0fbda82fb84730a663338f3552d36a at {username='slave'} at 127.27.17.129:41907: session id = 8b0fbda82fb84730a663338f3552d36a-b5bd5361adba43f09bdf5c51cda4bbcd
I20260810 22:04:13.748616 28384 tablet_copy_source_session.cc:205] T b5bd5361adba43f09bdf5c51cda4bbcd P 050f5ddd2d7c44c8a923f9f876d4e186: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:04:13.752120 28874 ts_tablet_manager.cc:1905] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260810 22:04:13.757064 28848 ts_tablet_manager.cc:1918] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown
I20260810 22:04:13.769590 28874 ts_tablet_manager.cc:1918] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown
I20260810 22:04:13.795279 28848 tablet_copy_client.cc:791] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: tablet copy: Starting download of 0 data blocks...
I20260810 22:04:13.796132 28848 tablet_copy_client.cc:655] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: tablet copy: Starting download of 1 WAL segments...
I20260810 22:04:13.796536 28874 tablet_copy_client.cc:791] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: tablet copy: Starting download of 0 data blocks...
I20260810 22:04:13.797434 28874 tablet_copy_client.cc:655] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: tablet copy: Starting download of 1 WAL segments...
I20260810 22:04:13.797632 28727 raft_consensus.cc:1203] T 312e5bba4bc240d39b6d15652a40cc79 P 8b0fbda82fb84730a663338f3552d36a [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.3->[2.4-2.4] Dedup: 2.4->[]
I20260810 22:04:13.805771 28848 tablet_copy_client.cc:531] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:04:13.806295 28874 tablet_copy_client.cc:531] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:04:13.815541 28848 tablet_bootstrap.cc:492] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: Bootstrap starting.
I20260810 22:04:13.816512 28874 tablet_bootstrap.cc:492] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: Bootstrap starting.
I20260810 22:04:13.857494 28874 tablet_bootstrap.cc:492] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:04:13.858455 28848 tablet_bootstrap.cc:492] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:04:13.858601 28874 tablet_bootstrap.cc:492] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: Bootstrap complete.
I20260810 22:04:13.859269 28874 ts_tablet_manager.cc:1397] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: Time spent bootstrapping tablet: real 0.043s user 0.021s sys 0.012s
I20260810 22:04:13.859450 28848 tablet_bootstrap.cc:492] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: Bootstrap complete.
I20260810 22:04:13.860018 28848 ts_tablet_manager.cc:1397] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: Time spent bootstrapping tablet: real 0.045s user 0.033s sys 0.005s
I20260810 22:04:13.862541 28848 raft_consensus.cc:348] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } }
I20260810 22:04:13.862543 28874 raft_consensus.cc:348] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } }
I20260810 22:04:13.863315 28848 raft_consensus.cc:729] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0fbda82fb84730a663338f3552d36a, State: Initialized, Role: FOLLOWER
I20260810 22:04:13.863332 28874 raft_consensus.cc:729] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0fbda82fb84730a663338f3552d36a, State: Initialized, Role: FOLLOWER
I20260810 22:04:13.864033 28848 consensus_queue.cc:260] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } }
I20260810 22:04:13.864025 28874 consensus_queue.cc:260] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9c63787081744750872196ba0013abed" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37287 } } peers { permanent_uuid: "050f5ddd2d7c44c8a923f9f876d4e186" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 36895 } } peers { permanent_uuid: "8b0fbda82fb84730a663338f3552d36a" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 41487 } }
I20260810 22:04:13.866266 28848 ts_tablet_manager.cc:1428] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260810 22:04:13.868091 28383 tablet_copy_service.cc:342] P 050f5ddd2d7c44c8a923f9f876d4e186: Request end of tablet copy session 8b0fbda82fb84730a663338f3552d36a-c849b79d77074d9eb6cfe7699ac8e78e received from {username='slave'} at 127.27.17.129:41907
I20260810 22:04:13.868451 28383 tablet_copy_service.cc:433] P 050f5ddd2d7c44c8a923f9f876d4e186: ending tablet copy session 8b0fbda82fb84730a663338f3552d36a-c849b79d77074d9eb6cfe7699ac8e78e on tablet c849b79d77074d9eb6cfe7699ac8e78e with peer 8b0fbda82fb84730a663338f3552d36a
I20260810 22:04:13.870810 28874 ts_tablet_manager.cc:1428] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260810 22:04:13.872473 28383 tablet_copy_service.cc:342] P 050f5ddd2d7c44c8a923f9f876d4e186: Request end of tablet copy session 8b0fbda82fb84730a663338f3552d36a-b5bd5361adba43f09bdf5c51cda4bbcd received from {username='slave'} at 127.27.17.129:41907
I20260810 22:04:13.872826 28383 tablet_copy_service.cc:433] P 050f5ddd2d7c44c8a923f9f876d4e186: ending tablet copy session 8b0fbda82fb84730a663338f3552d36a-b5bd5361adba43f09bdf5c51cda4bbcd on tablet b5bd5361adba43f09bdf5c51cda4bbcd with peer 8b0fbda82fb84730a663338f3552d36a
I20260810 22:04:13.918596 28317 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20260810 22:04:13.923170 28695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 22:04:13.925819 28132 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
9a1244ce7efc411c91289f1ecd847644 | 127.27.17.190:34181 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:33389 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
050f5ddd2d7c44c8a923f9f876d4e186 | 127.27.17.131:36895 | HEALTHY | <none> | 4 | 0
8b0fbda82fb84730a663338f3552d36a | 127.27.17.129:41487 | HEALTHY | <none> | 0 | 0
9c63787081744750872196ba0013abed | 127.27.17.130:37287 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.27.17.129:41487
env_inject_eio | 1 | hidden | 127.27.17.129:41487
env_inject_eio_globs | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1/** | hidden | 127.27.17.129:41487
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.27.17.129 | experimental | 127.27.17.129:41487
local_ip_for_outbound_sockets | 127.27.17.130 | experimental | 127.27.17.130:37287
local_ip_for_outbound_sockets | 127.27.17.131 | experimental | 127.27.17.131:36895
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.27.17.129:41487
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.27.17.130:37287
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.27.17.131:36895
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:33389 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet c849b79d77074d9eb6cfe7699ac8e78e of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
9c63787081744750872196ba0013abed (127.27.17.130:37287): RUNNING
050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131:36895): RUNNING [LEADER]
8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): RUNNING
All reported replicas are:
A = 9c63787081744750872196ba0013abed
B = 050f5ddd2d7c44c8a923f9f876d4e186
C = 8b0fbda82fb84730a663338f3552d36a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 6 | Yes
B | A B* C | 2 | 6 | Yes
C | A B C | 2 | 6 | Yes
Tablet b5bd5361adba43f09bdf5c51cda4bbcd of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
9c63787081744750872196ba0013abed (127.27.17.130:37287): RUNNING
050f5ddd2d7c44c8a923f9f876d4e186 (127.27.17.131:36895): RUNNING [LEADER]
8b0fbda82fb84730a663338f3552d36a (127.27.17.129:41487): RUNNING
All reported replicas are:
A = 9c63787081744750872196ba0013abed
B = 050f5ddd2d7c44c8a923f9f876d4e186
C = 8b0fbda82fb84730a663338f3552d36a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 5 | Yes
B | A B* C | 1 | 5 | Yes
C | A B C | 1 | 5 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 5 | 3 | 0 | 0 | 2
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 5
First Quartile | 5
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 22:04:14.134982 28727 raft_consensus.cc:1203] T b5bd5361adba43f09bdf5c51cda4bbcd P 8b0fbda82fb84730a663338f3552d36a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260810 22:04:14.138576 27718 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 22:04:14.195778 28727 raft_consensus.cc:1203] T c849b79d77074d9eb6cfe7699ac8e78e P 8b0fbda82fb84730a663338f3552d36a [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6] Dedup: 2.6->[]
I20260810 22:04:14.618428 28317 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20260810 22:04:14.619544 28695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 22:04:14.623379 28132 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
9a1244ce7efc411c91289f1ecd847644 | 127.27.17.190:34181 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:33389 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
050f5ddd2d7c44c8a923f9f876d4e186 | 127.27.17.131:36895 | HEALTHY | <none> | 4 | 0
8b0fbda82fb84730a663338f3552d36a | 127.27.17.129:41487 | HEALTHY | <none> | 0 | 0
9c63787081744750872196ba0013abed | 127.27.17.130:37287 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.27.17.129:41487
env_inject_eio | 1 | hidden | 127.27.17.129:41487
env_inject_eio_globs | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1/** | hidden | 127.27.17.129:41487
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.27.17.129 | experimental | 127.27.17.129:41487
local_ip_for_outbound_sockets | 127.27.17.130 | experimental | 127.27.17.130:37287
local_ip_for_outbound_sockets | 127.27.17.131 | experimental | 127.27.17.131:36895
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.27.17.129:41487
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.27.17.130:37287
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.27.17.131:36895
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:33389 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 5 | 5 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 5
First Quartile | 5
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260810 22:04:14.865752 27718 log_verifier.cc:126] Checking tablet 140448518535465eafa77555fa59db8c
I20260810 22:04:14.931941 27718 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 140448518535465eafa77555fa59db8c
I20260810 22:04:14.932225 27718 log_verifier.cc:126] Checking tablet 19b09a379fab4ad6aa1985c84fe6654a
I20260810 22:04:14.990801 27718 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 19b09a379fab4ad6aa1985c84fe6654a
I20260810 22:04:14.991017 27718 log_verifier.cc:126] Checking tablet 312e5bba4bc240d39b6d15652a40cc79
I20260810 22:04:15.056035 27718 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 312e5bba4bc240d39b6d15652a40cc79
I20260810 22:04:15.056257 27718 log_verifier.cc:126] Checking tablet b5bd5361adba43f09bdf5c51cda4bbcd
I20260810 22:04:15.123456 27718 log_verifier.cc:177] Verified matching terms for 5 ops in tablet b5bd5361adba43f09bdf5c51cda4bbcd
I20260810 22:04:15.123776 27718 log_verifier.cc:126] Checking tablet c849b79d77074d9eb6cfe7699ac8e78e
I20260810 22:04:15.209622 27718 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c849b79d77074d9eb6cfe7699ac8e78e
W20260810 22:04:15.751174 27718 scanner-internal.cc:458] Time spent opening tablet: real 0.513s user 0.011s sys 0.001s
I20260810 22:04:15.801714 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 28569
I20260810 22:04:15.853006 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 28044
I20260810 22:04:15.900486 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 28227
I20260810 22:04:15.948650 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 27730
2026-08-10T22:04:15Z chronyd exiting
[ OK ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/0 (16757 ms)
[ RUN ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/1
2026-08-10T22:04:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:04:16Z Disabled control of system clock
I20260810 22:04:16.078658 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.190:35505
--webserver_interface=127.27.17.190
--webserver_port=0
--builtin_ntp_servers=127.27.17.148:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.190:35505
--raft_prepare_replacement_before_eviction=true with env {}
W20260810 22:04:16.391911 28964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:16.392596 28964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:16.393090 28964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:16.424286 28964 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20260810 22:04:16.424747 28964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:04:16.425037 28964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:16.425312 28964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:04:16.425599 28964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:04:16.461377 28964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.190:35505
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.190:35505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
--webserver_interface=127.27.17.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:16.462731 28964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:04:16.464357 28964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:16.481904 28972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:16.481918 28977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:16.484221 28964 server_base.cc:1034] running on GCE node
W20260810 22:04:16.482713 28974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:17.673087 28964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:17.675730 28964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:17.677084 28964 hybrid_clock.cc:648] HybridClock initialized: now 1786399457677045 us; error 64 us; skew 500 ppm
I20260810 22:04:17.678066 28964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:17.685528 28964 webserver.cc:460] Webserver started at http://127.27.17.190:33621/ using document root <none> and password file <none>
I20260810 22:04:17.687084 28964 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:17.687332 28964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:04:17.688023 28964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:04:17.693178 28964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-0/instance:
uuid: "d7577a8b027a4a80ad0f7b2f460658b7"
format_stamp: "Formatted at 2026-08-10 22:04:17 on dist-test-slave-bw88"
I20260810 22:04:17.694461 28964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-1/instance:
uuid: "d7577a8b027a4a80ad0f7b2f460658b7"
format_stamp: "Formatted at 2026-08-10 22:04:17 on dist-test-slave-bw88"
I20260810 22:04:17.695432 28964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-2/instance:
uuid: "d7577a8b027a4a80ad0f7b2f460658b7"
format_stamp: "Formatted at 2026-08-10 22:04:17 on dist-test-slave-bw88"
I20260810 22:04:17.696338 28964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-3/instance:
uuid: "d7577a8b027a4a80ad0f7b2f460658b7"
format_stamp: "Formatted at 2026-08-10 22:04:17 on dist-test-slave-bw88"
I20260810 22:04:17.697219 28964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-4/instance:
uuid: "d7577a8b027a4a80ad0f7b2f460658b7"
format_stamp: "Formatted at 2026-08-10 22:04:17 on dist-test-slave-bw88"
I20260810 22:04:17.698173 28964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/wal/instance:
uuid: "d7577a8b027a4a80ad0f7b2f460658b7"
format_stamp: "Formatted at 2026-08-10 22:04:17 on dist-test-slave-bw88"
I20260810 22:04:17.718390 28964 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.014s sys 0.006s
I20260810 22:04:17.722913 28964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:17.723141 28964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/wal
uuid: "d7577a8b027a4a80ad0f7b2f460658b7"
format_stamp: "Formatted at 2026-08-10 22:04:17 on dist-test-slave-bw88"
I20260810 22:04:17.723497 28964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260810 22:04:17.765431 28964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:17.766948 28964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:17.767390 28964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:17.838668 28964 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.190:35505
I20260810 22:04:17.839174 29059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.190:35505 every 8 connection(s)
I20260810 22:04:17.841609 28964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
I20260810 22:04:17.842240 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 28964
I20260810 22:04:17.842669 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/wal/instance
I20260810 22:04:17.871747 29060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7: Bootstrap starting.
I20260810 22:04:17.878142 29060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:17.879940 29060 log.cc:824] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:17.884763 29060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7: No bootstrap required, opened a new log
I20260810 22:04:17.905092 29060 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7577a8b027a4a80ad0f7b2f460658b7" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 35505 } }
I20260810 22:04:17.905893 29060 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:17.906361 29060 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7577a8b027a4a80ad0f7b2f460658b7, State: Initialized, Role: FOLLOWER
I20260810 22:04:17.907186 29060 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7577a8b027a4a80ad0f7b2f460658b7" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 35505 } }
I20260810 22:04:17.907728 29060 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:04:17.908013 29060 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:04:17.908349 29060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:17.912318 29060 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7577a8b027a4a80ad0f7b2f460658b7" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 35505 } }
I20260810 22:04:17.913045 29060 leader_election.cc:302] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7577a8b027a4a80ad0f7b2f460658b7; no voters:
I20260810 22:04:17.914963 29060 leader_election.cc:288] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:04:17.915452 29066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:17.918061 29066 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7 [term 1 LEADER]: Becoming Leader. State: Replica: d7577a8b027a4a80ad0f7b2f460658b7, State: Running, Role: LEADER
I20260810 22:04:17.918810 29066 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7577a8b027a4a80ad0f7b2f460658b7" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 35505 } }
I20260810 22:04:17.919757 29060 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:04:17.930863 29067 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7577a8b027a4a80ad0f7b2f460658b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7577a8b027a4a80ad0f7b2f460658b7" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 35505 } } }
I20260810 22:04:17.931746 29067 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7 [sys.catalog]: This master's current role is: LEADER
I20260810 22:04:17.933596 29068 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7577a8b027a4a80ad0f7b2f460658b7. Latest consensus state: current_term: 1 leader_uuid: "d7577a8b027a4a80ad0f7b2f460658b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7577a8b027a4a80ad0f7b2f460658b7" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 35505 } } }
I20260810 22:04:17.934494 29068 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7 [sys.catalog]: This master's current role is: LEADER
I20260810 22:04:17.935847 29076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:04:17.954366 29076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:04:17.969633 29076 catalog_manager.cc:1347] Generated new cluster ID: 888bec2db48e4227bed800bcedf0a51e
I20260810 22:04:17.969970 29076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:04:17.996644 29076 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:04:17.998684 29076 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:04:18.018561 29076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7577a8b027a4a80ad0f7b2f460658b7: Generated new TSK 0
I20260810 22:04:18.019701 29076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:04:18.044030 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.129:0
--local_ip_for_outbound_sockets=127.27.17.129
--webserver_interface=127.27.17.129
--webserver_port=0
--tserver_master_addrs=127.27.17.190:35505
--builtin_ntp_servers=127.27.17.148:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20260810 22:04:18.359690 29088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:18.360240 29088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:18.360754 29088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:18.391362 29088 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20260810 22:04:18.391794 29088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:18.392591 29088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.129
I20260810 22:04:18.426980 29088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.27.17.129
--webserver_port=0
--tserver_master_addrs=127.27.17.190:35505
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.129
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:18.428277 29088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:04:18.429942 29088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:18.451277 29100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:18.453708 29104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:19.650264 29102 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 22:04:19.650300 29088 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260810 22:04:18.451622 29095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:19.654943 29088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:19.657809 29088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:19.659271 29088 hybrid_clock.cc:648] HybridClock initialized: now 1786399459659217 us; error 64 us; skew 500 ppm
I20260810 22:04:19.660297 29088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:19.671140 29088 webserver.cc:460] Webserver started at http://127.27.17.129:38101/ using document root <none> and password file <none>
I20260810 22:04:19.673183 29088 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:19.673502 29088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:04:19.674376 29088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:04:19.681519 29088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-0/instance:
uuid: "9eb15a3b74af456588e8733e2406f8ed"
format_stamp: "Formatted at 2026-08-10 22:04:19 on dist-test-slave-bw88"
I20260810 22:04:19.682862 29088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-1/instance:
uuid: "9eb15a3b74af456588e8733e2406f8ed"
format_stamp: "Formatted at 2026-08-10 22:04:19 on dist-test-slave-bw88"
I20260810 22:04:19.683776 29088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-2/instance:
uuid: "9eb15a3b74af456588e8733e2406f8ed"
format_stamp: "Formatted at 2026-08-10 22:04:19 on dist-test-slave-bw88"
I20260810 22:04:19.684669 29088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-3/instance:
uuid: "9eb15a3b74af456588e8733e2406f8ed"
format_stamp: "Formatted at 2026-08-10 22:04:19 on dist-test-slave-bw88"
I20260810 22:04:19.685531 29088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-4/instance:
uuid: "9eb15a3b74af456588e8733e2406f8ed"
format_stamp: "Formatted at 2026-08-10 22:04:19 on dist-test-slave-bw88"
I20260810 22:04:19.686326 29088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/wal/instance:
uuid: "9eb15a3b74af456588e8733e2406f8ed"
format_stamp: "Formatted at 2026-08-10 22:04:19 on dist-test-slave-bw88"
I20260810 22:04:19.706398 29088 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.011s sys 0.008s
I20260810 22:04:19.710863 29088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:19.711081 29088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/wal
uuid: "9eb15a3b74af456588e8733e2406f8ed"
format_stamp: "Formatted at 2026-08-10 22:04:19 on dist-test-slave-bw88"
I20260810 22:04:19.711416 29088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260810 22:04:19.761093 29088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:19.762655 29088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:19.763065 29088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:19.765712 29088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:04:19.770320 29088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:04:19.770560 29088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:19.770836 29088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:04:19.771006 29088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:19.923952 29088 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.129:37957
I20260810 22:04:19.924492 29275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.129:37957 every 8 connection(s)
I20260810 22:04:19.926776 29088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
I20260810 22:04:19.932742 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 29088
I20260810 22:04:19.933851 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/wal/instance
I20260810 22:04:19.943347 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.130:0
--local_ip_for_outbound_sockets=127.27.17.130
--webserver_interface=127.27.17.130
--webserver_port=0
--tserver_master_addrs=127.27.17.190:35505
--builtin_ntp_servers=127.27.17.148:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260810 22:04:19.956743 29279 heartbeater.cc:347] Connected to a master server at 127.27.17.190:35505
I20260810 22:04:19.957290 29279 heartbeater.cc:464] Registering TS with master...
I20260810 22:04:19.958694 29279 heartbeater.cc:511] Master 127.27.17.190:35505 requested a full tablet report, sending...
I20260810 22:04:19.962049 29015 ts_manager.cc:194] Registered new tserver with Master: 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957)
I20260810 22:04:19.965406 29015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.129:59925
W20260810 22:04:20.264006 29286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:20.264506 29286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:20.264946 29286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:20.295914 29286 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20260810 22:04:20.296274 29286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:20.297025 29286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.130
I20260810 22:04:20.339596 29286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
--webserver_interface=127.27.17.130
--webserver_port=0
--tserver_master_addrs=127.27.17.190:35505
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.130
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:20.340807 29286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:04:20.342540 29286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:20.361723 29293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:20.970960 29279 heartbeater.cc:503] Master 127.27.17.190:35505 was elected leader, sending a full tablet report...
W20260810 22:04:20.361747 29294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:20.364319 29286 server_base.cc:1034] running on GCE node
W20260810 22:04:20.362476 29296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:21.612272 29286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:21.614845 29286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:21.616376 29286 hybrid_clock.cc:648] HybridClock initialized: now 1786399461616336 us; error 44 us; skew 500 ppm
I20260810 22:04:21.617224 29286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:21.624635 29286 webserver.cc:460] Webserver started at http://127.27.17.130:42389/ using document root <none> and password file <none>
I20260810 22:04:21.626260 29286 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:21.626511 29286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:04:21.627228 29286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:04:21.633072 29286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-0/instance:
uuid: "6b67f5fd3c9943a7bf77411798547a55"
format_stamp: "Formatted at 2026-08-10 22:04:21 on dist-test-slave-bw88"
I20260810 22:04:21.634330 29286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-1/instance:
uuid: "6b67f5fd3c9943a7bf77411798547a55"
format_stamp: "Formatted at 2026-08-10 22:04:21 on dist-test-slave-bw88"
I20260810 22:04:21.635409 29286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-2/instance:
uuid: "6b67f5fd3c9943a7bf77411798547a55"
format_stamp: "Formatted at 2026-08-10 22:04:21 on dist-test-slave-bw88"
I20260810 22:04:21.636345 29286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-3/instance:
uuid: "6b67f5fd3c9943a7bf77411798547a55"
format_stamp: "Formatted at 2026-08-10 22:04:21 on dist-test-slave-bw88"
I20260810 22:04:21.637291 29286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-4/instance:
uuid: "6b67f5fd3c9943a7bf77411798547a55"
format_stamp: "Formatted at 2026-08-10 22:04:21 on dist-test-slave-bw88"
I20260810 22:04:21.638171 29286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/wal/instance:
uuid: "6b67f5fd3c9943a7bf77411798547a55"
format_stamp: "Formatted at 2026-08-10 22:04:21 on dist-test-slave-bw88"
I20260810 22:04:21.658221 29286 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.012s sys 0.007s
I20260810 22:04:21.662708 29286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:21.662993 29286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/wal
uuid: "6b67f5fd3c9943a7bf77411798547a55"
format_stamp: "Formatted at 2026-08-10 22:04:21 on dist-test-slave-bw88"
I20260810 22:04:21.663347 29286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260810 22:04:21.707706 29286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:21.709197 29286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:21.709697 29286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:21.712992 29286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:04:21.717835 29286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:04:21.718034 29286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:21.718333 29286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:04:21.718508 29286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:21.874677 29286 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.130:44819
I20260810 22:04:21.875178 29474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.130:44819 every 8 connection(s)
I20260810 22:04:21.877485 29286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
I20260810 22:04:21.888155 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 29286
I20260810 22:04:21.888674 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/wal/instance
I20260810 22:04:21.895735 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.131:0
--local_ip_for_outbound_sockets=127.27.17.131
--webserver_interface=127.27.17.131
--webserver_port=0
--tserver_master_addrs=127.27.17.190:35505
--builtin_ntp_servers=127.27.17.148:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260810 22:04:21.906045 29475 heartbeater.cc:347] Connected to a master server at 127.27.17.190:35505
I20260810 22:04:21.906512 29475 heartbeater.cc:464] Registering TS with master...
I20260810 22:04:21.907793 29475 heartbeater.cc:511] Master 127.27.17.190:35505 requested a full tablet report, sending...
I20260810 22:04:21.910334 29015 ts_manager.cc:194] Registered new tserver with Master: 6b67f5fd3c9943a7bf77411798547a55 (127.27.17.130:44819)
I20260810 22:04:21.911832 29015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.130:52521
W20260810 22:04:22.219993 29481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:22.220611 29481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:22.221148 29481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:22.252460 29481 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20260810 22:04:22.252915 29481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:22.253799 29481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.131
I20260810 22:04:22.289091 29481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
--webserver_interface=127.27.17.131
--webserver_port=0
--tserver_master_addrs=127.27.17.190:35505
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.131
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:22.290362 29481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:04:22.292058 29481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:04:22.915592 29475 heartbeater.cc:503] Master 127.27.17.190:35505 was elected leader, sending a full tablet report...
W20260810 22:04:22.310793 29487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:22.310755 29492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:22.310807 29488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:23.487751 29489 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 22:04:23.487908 29481 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 22:04:23.491889 29481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:23.494752 29481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:23.496199 29481 hybrid_clock.cc:648] HybridClock initialized: now 1786399463496140 us; error 81 us; skew 500 ppm
I20260810 22:04:23.496987 29481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:23.504069 29481 webserver.cc:460] Webserver started at http://127.27.17.131:39527/ using document root <none> and password file <none>
I20260810 22:04:23.505625 29481 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:23.505899 29481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:04:23.506579 29481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:04:23.511595 29481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-0/instance:
uuid: "cfe74abd0282488f9a8ce6c152b761b2"
format_stamp: "Formatted at 2026-08-10 22:04:23 on dist-test-slave-bw88"
I20260810 22:04:23.512773 29481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-1/instance:
uuid: "cfe74abd0282488f9a8ce6c152b761b2"
format_stamp: "Formatted at 2026-08-10 22:04:23 on dist-test-slave-bw88"
I20260810 22:04:23.514014 29481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-2/instance:
uuid: "cfe74abd0282488f9a8ce6c152b761b2"
format_stamp: "Formatted at 2026-08-10 22:04:23 on dist-test-slave-bw88"
I20260810 22:04:23.514873 29481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-3/instance:
uuid: "cfe74abd0282488f9a8ce6c152b761b2"
format_stamp: "Formatted at 2026-08-10 22:04:23 on dist-test-slave-bw88"
I20260810 22:04:23.515717 29481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-4/instance:
uuid: "cfe74abd0282488f9a8ce6c152b761b2"
format_stamp: "Formatted at 2026-08-10 22:04:23 on dist-test-slave-bw88"
I20260810 22:04:23.516533 29481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/wal/instance:
uuid: "cfe74abd0282488f9a8ce6c152b761b2"
format_stamp: "Formatted at 2026-08-10 22:04:23 on dist-test-slave-bw88"
I20260810 22:04:23.537161 29481 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.011s sys 0.011s
I20260810 22:04:23.542237 29481 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:23.542476 29481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/wal
uuid: "cfe74abd0282488f9a8ce6c152b761b2"
format_stamp: "Formatted at 2026-08-10 22:04:23 on dist-test-slave-bw88"
I20260810 22:04:23.542825 29481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260810 22:04:23.579180 29481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:23.580646 29481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:23.581086 29481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:23.583983 29481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:04:23.588672 29481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:04:23.588867 29481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:23.589183 29481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:04:23.589327 29481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:23.734323 29481 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.131:38077
I20260810 22:04:23.734759 29672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.131:38077 every 8 connection(s)
I20260810 22:04:23.737200 29481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
I20260810 22:04:23.743163 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 29481
I20260810 22:04:23.743669 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/wal/instance
I20260810 22:04:23.750427 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.132:0
--local_ip_for_outbound_sockets=127.27.17.132
--webserver_interface=127.27.17.132
--webserver_port=0
--tserver_master_addrs=127.27.17.190:35505
--builtin_ntp_servers=127.27.17.148:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260810 22:04:23.762169 29674 heartbeater.cc:347] Connected to a master server at 127.27.17.190:35505
I20260810 22:04:23.762677 29674 heartbeater.cc:464] Registering TS with master...
I20260810 22:04:23.763741 29674 heartbeater.cc:511] Master 127.27.17.190:35505 requested a full tablet report, sending...
I20260810 22:04:23.766145 29015 ts_manager.cc:194] Registered new tserver with Master: cfe74abd0282488f9a8ce6c152b761b2 (127.27.17.131:38077)
I20260810 22:04:23.767831 29015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.131:50137
W20260810 22:04:24.102003 29681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:24.102586 29681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:24.103099 29681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:24.135118 29681 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20260810 22:04:24.135576 29681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:24.136471 29681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.132
I20260810 22:04:24.171656 29681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-0/info.pb
--webserver_interface=127.27.17.132
--webserver_port=0
--tserver_master_addrs=127.27.17.190:35505
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.132
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:24.172971 29681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:04:24.174925 29681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:24.195464 29691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:24.771607 29674 heartbeater.cc:503] Master 127.27.17.190:35505 was elected leader, sending a full tablet report...
W20260810 22:04:24.195530 29696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:24.197751 29681 server_base.cc:1034] running on GCE node
W20260810 22:04:24.196260 29692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:25.414196 29681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:25.417266 29681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:25.418779 29681 hybrid_clock.cc:648] HybridClock initialized: now 1786399465418742 us; error 54 us; skew 500 ppm
I20260810 22:04:25.419627 29681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:25.435081 29681 webserver.cc:460] Webserver started at http://127.27.17.132:41559/ using document root <none> and password file <none>
I20260810 22:04:25.436765 29681 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:25.437127 29681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:04:25.437896 29681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:04:25.443015 29681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-0/instance:
uuid: "21ef9e69018e484a8dbf082e54bbc07a"
format_stamp: "Formatted at 2026-08-10 22:04:25 on dist-test-slave-bw88"
I20260810 22:04:25.444257 29681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-1/instance:
uuid: "21ef9e69018e484a8dbf082e54bbc07a"
format_stamp: "Formatted at 2026-08-10 22:04:25 on dist-test-slave-bw88"
I20260810 22:04:25.445225 29681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-2/instance:
uuid: "21ef9e69018e484a8dbf082e54bbc07a"
format_stamp: "Formatted at 2026-08-10 22:04:25 on dist-test-slave-bw88"
I20260810 22:04:25.446175 29681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-3/instance:
uuid: "21ef9e69018e484a8dbf082e54bbc07a"
format_stamp: "Formatted at 2026-08-10 22:04:25 on dist-test-slave-bw88"
I20260810 22:04:25.447064 29681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-4/instance:
uuid: "21ef9e69018e484a8dbf082e54bbc07a"
format_stamp: "Formatted at 2026-08-10 22:04:25 on dist-test-slave-bw88"
I20260810 22:04:25.447929 29681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/wal/instance:
uuid: "21ef9e69018e484a8dbf082e54bbc07a"
format_stamp: "Formatted at 2026-08-10 22:04:25 on dist-test-slave-bw88"
I20260810 22:04:25.467878 29681 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.011s sys 0.010s
I20260810 22:04:25.472353 29681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:25.472626 29681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/wal
uuid: "21ef9e69018e484a8dbf082e54bbc07a"
format_stamp: "Formatted at 2026-08-10 22:04:25 on dist-test-slave-bw88"
I20260810 22:04:25.473023 29681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260810 22:04:25.522215 29681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:25.524165 29681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:25.524645 29681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:25.527333 29681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:04:25.532106 29681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:04:25.532338 29681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:25.532635 29681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:04:25.532848 29681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:25.682309 29681 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.132:38435
I20260810 22:04:25.682804 29857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.132:38435 every 8 connection(s)
I20260810 22:04:25.685222 29681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-0/info.pb
I20260810 22:04:25.688796 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 29681
I20260810 22:04:25.689327 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/wal/instance
I20260810 22:04:25.711431 29859 heartbeater.cc:347] Connected to a master server at 127.27.17.190:35505
I20260810 22:04:25.711879 29859 heartbeater.cc:464] Registering TS with master...
I20260810 22:04:25.712796 29859 heartbeater.cc:511] Master 127.27.17.190:35505 requested a full tablet report, sending...
I20260810 22:04:25.715044 29014 ts_manager.cc:194] Registered new tserver with Master: 21ef9e69018e484a8dbf082e54bbc07a (127.27.17.132:38435)
I20260810 22:04:25.716663 29014 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.132:55949
I20260810 22:04:25.726110 27718 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20260810 22:04:25.762359 27718 test_util.cc:268] Using random seed: -1986695648
I20260810 22:04:25.805833 29014 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38692:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\231\231\231\031\004\001\0002333\004\001\000\313\314\314L\004\001\000dfff"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 22:04:25.895927 29576 tablet_service.cc:1467] Processing CreateTablet for tablet cabe053775754da49c6bc46e79d13bf3 (DEFAULT_TABLE table=test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260810 22:04:25.910729 29374 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba3cc9bd6c9412dba778d21fd65181c (DEFAULT_TABLE table=test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260810 22:04:25.914557 29373 tablet_service.cc:1467] Processing CreateTablet for tablet fff845c32f1f4b81b6b8d0052cf06968 (DEFAULT_TABLE table=test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260810 22:04:25.917276 29575 tablet_service.cc:1467] Processing CreateTablet for tablet fff845c32f1f4b81b6b8d0052cf06968 (DEFAULT_TABLE table=test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260810 22:04:25.917973 29372 tablet_service.cc:1467] Processing CreateTablet for tablet dcfa54f1b7ef4b0b95226226bd78c40b (DEFAULT_TABLE table=test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260810 22:04:25.919184 29577 tablet_service.cc:1467] Processing CreateTablet for tablet 8690baf5d2f54129846bba669f5c6f3d (DEFAULT_TABLE table=test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260810 22:04:25.920188 29572 tablet_service.cc:1467] Processing CreateTablet for tablet dcfa54f1b7ef4b0b95226226bd78c40b (DEFAULT_TABLE table=test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260810 22:04:25.908776 29375 tablet_service.cc:1467] Processing CreateTablet for tablet 8690baf5d2f54129846bba669f5c6f3d (DEFAULT_TABLE table=test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260810 22:04:25.938792 29186 tablet_service.cc:1467] Processing CreateTablet for tablet 8690baf5d2f54129846bba669f5c6f3d (DEFAULT_TABLE table=test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260810 22:04:25.939858 29182 tablet_service.cc:1467] Processing CreateTablet for tablet dcfa54f1b7ef4b0b95226226bd78c40b (DEFAULT_TABLE table=test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260810 22:04:25.941634 29184 tablet_service.cc:1467] Processing CreateTablet for tablet cabe053775754da49c6bc46e79d13bf3 (DEFAULT_TABLE table=test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260810 22:04:25.942216 29185 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba3cc9bd6c9412dba778d21fd65181c (DEFAULT_TABLE table=test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260810 22:04:25.970703 29771 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba3cc9bd6c9412dba778d21fd65181c (DEFAULT_TABLE table=test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260810 22:04:25.984215 29770 tablet_service.cc:1467] Processing CreateTablet for tablet cabe053775754da49c6bc46e79d13bf3 (DEFAULT_TABLE table=test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260810 22:04:25.990943 29769 tablet_service.cc:1467] Processing CreateTablet for tablet fff845c32f1f4b81b6b8d0052cf06968 (DEFAULT_TABLE table=test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260810 22:04:26.030835 29896 tablet_bootstrap.cc:492] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55: Bootstrap starting.
I20260810 22:04:26.033849 29897 tablet_bootstrap.cc:492] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55: Bootstrap starting.
I20260810 22:04:26.045023 29898 tablet_bootstrap.cc:492] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55: Bootstrap starting.
I20260810 22:04:26.053978 29894 tablet_bootstrap.cc:492] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2: Bootstrap starting.
I20260810 22:04:26.055250 29899 tablet_bootstrap.cc:492] T fff845c32f1f4b81b6b8d0052cf06968 P 21ef9e69018e484a8dbf082e54bbc07a: Bootstrap starting.
I20260810 22:04:26.062673 29895 tablet_bootstrap.cc:492] T fff845c32f1f4b81b6b8d0052cf06968 P cfe74abd0282488f9a8ce6c152b761b2: Bootstrap starting.
I20260810 22:04:26.071458 29897 tablet_bootstrap.cc:654] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:26.073544 29899 tablet_bootstrap.cc:654] T fff845c32f1f4b81b6b8d0052cf06968 P 21ef9e69018e484a8dbf082e54bbc07a: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:26.077944 29902 tablet_bootstrap.cc:492] T 7ba3cc9bd6c9412dba778d21fd65181c P 21ef9e69018e484a8dbf082e54bbc07a: Bootstrap starting.
I20260810 22:04:26.078732 29903 tablet_bootstrap.cc:492] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed: Bootstrap starting.
I20260810 22:04:26.080209 29899 log.cc:824] T fff845c32f1f4b81b6b8d0052cf06968 P 21ef9e69018e484a8dbf082e54bbc07a: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:26.085495 29904 tablet_bootstrap.cc:492] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed: Bootstrap starting.
I20260810 22:04:26.089843 29898 tablet_bootstrap.cc:654] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:26.069205 29896 tablet_bootstrap.cc:654] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:26.092352 29896 log.cc:824] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:26.094616 29905 tablet_bootstrap.cc:492] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55: Bootstrap starting.
I20260810 22:04:26.092829 29899 tablet_bootstrap.cc:492] T fff845c32f1f4b81b6b8d0052cf06968 P 21ef9e69018e484a8dbf082e54bbc07a: No bootstrap required, opened a new log
I20260810 22:04:26.098004 29899 ts_tablet_manager.cc:1397] T fff845c32f1f4b81b6b8d0052cf06968 P 21ef9e69018e484a8dbf082e54bbc07a: Time spent bootstrapping tablet: real 0.043s user 0.016s sys 0.004s
I20260810 22:04:26.098038 29893 tablet_bootstrap.cc:492] T dcfa54f1b7ef4b0b95226226bd78c40b P cfe74abd0282488f9a8ce6c152b761b2: Bootstrap starting.
I20260810 22:04:26.097743 29901 tablet_bootstrap.cc:492] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a: Bootstrap starting.
I20260810 22:04:26.099345 29904 tablet_bootstrap.cc:654] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:26.104664 29902 tablet_bootstrap.cc:654] T 7ba3cc9bd6c9412dba778d21fd65181c P 21ef9e69018e484a8dbf082e54bbc07a: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:26.105151 29901 tablet_bootstrap.cc:654] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:26.106568 29907 tablet_bootstrap.cc:492] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed: Bootstrap starting.
I20260810 22:04:26.112933 29905 tablet_bootstrap.cc:654] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:26.114183 29904 log.cc:824] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:26.127934 29894 tablet_bootstrap.cc:654] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:26.134758 29907 tablet_bootstrap.cc:654] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:26.140301 29893 tablet_bootstrap.cc:654] T dcfa54f1b7ef4b0b95226226bd78c40b P cfe74abd0282488f9a8ce6c152b761b2: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:26.141690 29903 tablet_bootstrap.cc:654] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:26.142386 29894 log.cc:824] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:26.156013 29897 tablet_bootstrap.cc:492] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55: No bootstrap required, opened a new log
I20260810 22:04:26.156013 29907 tablet_bootstrap.cc:492] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed: No bootstrap required, opened a new log
I20260810 22:04:26.156576 29897 ts_tablet_manager.cc:1397] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55: Time spent bootstrapping tablet: real 0.123s user 0.016s sys 0.018s
I20260810 22:04:26.156571 29907 ts_tablet_manager.cc:1397] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed: Time spent bootstrapping tablet: real 0.050s user 0.014s sys 0.000s
I20260810 22:04:26.161645 29896 tablet_bootstrap.cc:492] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55: No bootstrap required, opened a new log
I20260810 22:04:26.162247 29896 ts_tablet_manager.cc:1397] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55: Time spent bootstrapping tablet: real 0.132s user 0.013s sys 0.030s
I20260810 22:04:26.161676 29898 tablet_bootstrap.cc:492] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55: No bootstrap required, opened a new log
I20260810 22:04:26.162736 29899 raft_consensus.cc:348] T fff845c32f1f4b81b6b8d0052cf06968 P 21ef9e69018e484a8dbf082e54bbc07a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } }
I20260810 22:04:26.163986 29898 ts_tablet_manager.cc:1397] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55: Time spent bootstrapping tablet: real 0.119s user 0.015s sys 0.000s
I20260810 22:04:26.163894 29899 raft_consensus.cc:374] T fff845c32f1f4b81b6b8d0052cf06968 P 21ef9e69018e484a8dbf082e54bbc07a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:26.164273 29899 raft_consensus.cc:729] T fff845c32f1f4b81b6b8d0052cf06968 P 21ef9e69018e484a8dbf082e54bbc07a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21ef9e69018e484a8dbf082e54bbc07a, State: Initialized, Role: FOLLOWER
I20260810 22:04:26.165748 29895 tablet_bootstrap.cc:654] T fff845c32f1f4b81b6b8d0052cf06968 P cfe74abd0282488f9a8ce6c152b761b2: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:26.165552 29899 consensus_queue.cc:260] T fff845c32f1f4b81b6b8d0052cf06968 P 21ef9e69018e484a8dbf082e54bbc07a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } }
I20260810 22:04:26.168553 29903 tablet_bootstrap.cc:492] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed: No bootstrap required, opened a new log
I20260810 22:04:26.169003 29903 ts_tablet_manager.cc:1397] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed: Time spent bootstrapping tablet: real 0.091s user 0.013s sys 0.002s
I20260810 22:04:26.171404 29899 ts_tablet_manager.cc:1428] T fff845c32f1f4b81b6b8d0052cf06968 P 21ef9e69018e484a8dbf082e54bbc07a: Time spent starting tablet: real 0.073s user 0.045s sys 0.000s
I20260810 22:04:26.172086 29904 tablet_bootstrap.cc:492] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed: No bootstrap required, opened a new log
I20260810 22:04:26.172405 29859 heartbeater.cc:503] Master 127.27.17.190:35505 was elected leader, sending a full tablet report...
I20260810 22:04:26.172562 29904 ts_tablet_manager.cc:1397] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed: Time spent bootstrapping tablet: real 0.088s user 0.005s sys 0.009s
I20260810 22:04:26.173236 29913 tablet_bootstrap.cc:492] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2: Bootstrap starting.
I20260810 22:04:26.187081 29913 tablet_bootstrap.cc:654] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2: Neither blocks nor log segments found. Creating new log.
W20260810 22:04:26.191634 29861 tablet.cc:2369] T fff845c32f1f4b81b6b8d0052cf06968 P 21ef9e69018e484a8dbf082e54bbc07a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:04:26.195571 29893 tablet_bootstrap.cc:492] T dcfa54f1b7ef4b0b95226226bd78c40b P cfe74abd0282488f9a8ce6c152b761b2: No bootstrap required, opened a new log
I20260810 22:04:26.198114 29901 tablet_bootstrap.cc:492] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a: No bootstrap required, opened a new log
I20260810 22:04:26.198462 29902 tablet_bootstrap.cc:492] T 7ba3cc9bd6c9412dba778d21fd65181c P 21ef9e69018e484a8dbf082e54bbc07a: No bootstrap required, opened a new log
I20260810 22:04:26.198541 29901 ts_tablet_manager.cc:1397] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a: Time spent bootstrapping tablet: real 0.101s user 0.006s sys 0.006s
I20260810 22:04:26.198838 29902 ts_tablet_manager.cc:1397] T 7ba3cc9bd6c9412dba778d21fd65181c P 21ef9e69018e484a8dbf082e54bbc07a: Time spent bootstrapping tablet: real 0.121s user 0.004s sys 0.012s
I20260810 22:04:26.201048 29912 tablet_bootstrap.cc:492] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed: Bootstrap starting.
I20260810 22:04:26.201303 29902 raft_consensus.cc:348] T 7ba3cc9bd6c9412dba778d21fd65181c P 21ef9e69018e484a8dbf082e54bbc07a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:26.202334 29902 raft_consensus.cc:374] T 7ba3cc9bd6c9412dba778d21fd65181c P 21ef9e69018e484a8dbf082e54bbc07a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:26.202451 29905 tablet_bootstrap.cc:492] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55: No bootstrap required, opened a new log
I20260810 22:04:26.202622 29902 raft_consensus.cc:729] T 7ba3cc9bd6c9412dba778d21fd65181c P 21ef9e69018e484a8dbf082e54bbc07a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21ef9e69018e484a8dbf082e54bbc07a, State: Initialized, Role: FOLLOWER
I20260810 22:04:26.202945 29905 ts_tablet_manager.cc:1397] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55: Time spent bootstrapping tablet: real 0.110s user 0.013s sys 0.000s
I20260810 22:04:26.203505 29893 ts_tablet_manager.cc:1397] T dcfa54f1b7ef4b0b95226226bd78c40b P cfe74abd0282488f9a8ce6c152b761b2: Time spent bootstrapping tablet: real 0.168s user 0.014s sys 0.008s
I20260810 22:04:26.206240 29902 consensus_queue.cc:260] T 7ba3cc9bd6c9412dba778d21fd65181c P 21ef9e69018e484a8dbf082e54bbc07a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:26.209781 29894 tablet_bootstrap.cc:492] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2: No bootstrap required, opened a new log
I20260810 22:04:26.210209 29894 ts_tablet_manager.cc:1397] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2: Time spent bootstrapping tablet: real 0.180s user 0.015s sys 0.005s
I20260810 22:04:26.216236 29905 raft_consensus.cc:348] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:26.218608 29902 ts_tablet_manager.cc:1428] T 7ba3cc9bd6c9412dba778d21fd65181c P 21ef9e69018e484a8dbf082e54bbc07a: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260810 22:04:26.218672 29895 tablet_bootstrap.cc:492] T fff845c32f1f4b81b6b8d0052cf06968 P cfe74abd0282488f9a8ce6c152b761b2: No bootstrap required, opened a new log
I20260810 22:04:26.217746 29898 raft_consensus.cc:348] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } }
I20260810 22:04:26.217751 29896 raft_consensus.cc:348] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:26.219063 29895 ts_tablet_manager.cc:1397] T fff845c32f1f4b81b6b8d0052cf06968 P cfe74abd0282488f9a8ce6c152b761b2: Time spent bootstrapping tablet: real 0.157s user 0.005s sys 0.008s
I20260810 22:04:26.219074 29898 raft_consensus.cc:374] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:26.219156 29896 raft_consensus.cc:374] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:26.219425 29898 raft_consensus.cc:729] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b67f5fd3c9943a7bf77411798547a55, State: Initialized, Role: FOLLOWER
I20260810 22:04:26.220553 29898 consensus_queue.cc:260] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } }
I20260810 22:04:26.222107 29901 raft_consensus.cc:348] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } }
I20260810 22:04:26.222855 29901 raft_consensus.cc:374] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:26.218129 29897 raft_consensus.cc:348] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } }
I20260810 22:04:26.223126 29901 raft_consensus.cc:729] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21ef9e69018e484a8dbf082e54bbc07a, State: Initialized, Role: FOLLOWER
I20260810 22:04:26.223635 29897 raft_consensus.cc:374] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:26.224110 29897 raft_consensus.cc:729] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b67f5fd3c9943a7bf77411798547a55, State: Initialized, Role: FOLLOWER
I20260810 22:04:26.223908 29901 consensus_queue.cc:260] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } }
I20260810 22:04:26.225291 29897 consensus_queue.cc:260] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } }
I20260810 22:04:26.228842 29912 tablet_bootstrap.cc:654] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:26.219470 29896 raft_consensus.cc:729] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b67f5fd3c9943a7bf77411798547a55, State: Initialized, Role: FOLLOWER
I20260810 22:04:26.229653 29896 consensus_queue.cc:260] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:26.226732 29905 raft_consensus.cc:374] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:26.231297 29905 raft_consensus.cc:729] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b67f5fd3c9943a7bf77411798547a55, State: Initialized, Role: FOLLOWER
I20260810 22:04:26.232379 29905 consensus_queue.cc:260] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:26.238310 29901 ts_tablet_manager.cc:1428] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a: Time spent starting tablet: real 0.039s user 0.006s sys 0.001s
I20260810 22:04:26.239801 29894 raft_consensus.cc:348] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } }
I20260810 22:04:26.240888 29894 raft_consensus.cc:374] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:26.241108 29898 ts_tablet_manager.cc:1428] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55: Time spent starting tablet: real 0.077s user 0.010s sys 0.004s
I20260810 22:04:26.241247 29894 raft_consensus.cc:729] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfe74abd0282488f9a8ce6c152b761b2, State: Initialized, Role: FOLLOWER
I20260810 22:04:26.242341 29913 tablet_bootstrap.cc:492] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2: No bootstrap required, opened a new log
I20260810 22:04:26.242769 29913 ts_tablet_manager.cc:1397] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2: Time spent bootstrapping tablet: real 0.096s user 0.013s sys 0.000s
I20260810 22:04:26.242882 29895 raft_consensus.cc:348] T fff845c32f1f4b81b6b8d0052cf06968 P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } }
I20260810 22:04:26.243769 29895 raft_consensus.cc:374] T fff845c32f1f4b81b6b8d0052cf06968 P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:26.244042 29895 raft_consensus.cc:729] T fff845c32f1f4b81b6b8d0052cf06968 P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfe74abd0282488f9a8ce6c152b761b2, State: Initialized, Role: FOLLOWER
I20260810 22:04:26.242343 29894 consensus_queue.cc:260] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } }
I20260810 22:04:26.247153 29895 consensus_queue.cc:260] T fff845c32f1f4b81b6b8d0052cf06968 P cfe74abd0282488f9a8ce6c152b761b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } }
I20260810 22:04:26.252496 29893 raft_consensus.cc:348] T dcfa54f1b7ef4b0b95226226bd78c40b P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } }
I20260810 22:04:26.252719 29904 raft_consensus.cc:348] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:26.253485 29893 raft_consensus.cc:374] T dcfa54f1b7ef4b0b95226226bd78c40b P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:26.253793 29904 raft_consensus.cc:374] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:26.253798 29893 raft_consensus.cc:729] T dcfa54f1b7ef4b0b95226226bd78c40b P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfe74abd0282488f9a8ce6c152b761b2, State: Initialized, Role: FOLLOWER
I20260810 22:04:26.254094 29904 raft_consensus.cc:729] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb15a3b74af456588e8733e2406f8ed, State: Initialized, Role: FOLLOWER
I20260810 22:04:26.254541 29893 consensus_queue.cc:260] T dcfa54f1b7ef4b0b95226226bd78c40b P cfe74abd0282488f9a8ce6c152b761b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } }
I20260810 22:04:26.255029 29904 consensus_queue.cc:260] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:26.257663 29903 raft_consensus.cc:348] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } }
I20260810 22:04:26.258433 29903 raft_consensus.cc:374] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:26.258567 29905 ts_tablet_manager.cc:1428] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55: Time spent starting tablet: real 0.055s user 0.014s sys 0.000s
I20260810 22:04:26.258744 29903 raft_consensus.cc:729] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb15a3b74af456588e8733e2406f8ed, State: Initialized, Role: FOLLOWER
I20260810 22:04:26.259456 29903 consensus_queue.cc:260] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } }
I20260810 22:04:26.267311 29907 raft_consensus.cc:348] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:26.271685 29896 ts_tablet_manager.cc:1428] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55: Time spent starting tablet: real 0.109s user 0.027s sys 0.009s
I20260810 22:04:26.272130 29907 raft_consensus.cc:374] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:26.276309 29907 raft_consensus.cc:729] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb15a3b74af456588e8733e2406f8ed, State: Initialized, Role: FOLLOWER
I20260810 22:04:26.275420 29903 ts_tablet_manager.cc:1428] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed: Time spent starting tablet: real 0.106s user 0.009s sys 0.012s
I20260810 22:04:26.291776 29940 raft_consensus.cc:482] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:26.298060 29897 ts_tablet_manager.cc:1428] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55: Time spent starting tablet: real 0.141s user 0.031s sys 0.019s
I20260810 22:04:26.297888 29907 consensus_queue.cc:260] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:26.298214 29940 raft_consensus.cc:504] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:26.301019 29907 ts_tablet_manager.cc:1428] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed: Time spent starting tablet: real 0.144s user 0.029s sys 0.019s
I20260810 22:04:26.303063 29895 ts_tablet_manager.cc:1428] T fff845c32f1f4b81b6b8d0052cf06968 P cfe74abd0282488f9a8ce6c152b761b2: Time spent starting tablet: real 0.084s user 0.016s sys 0.000s
I20260810 22:04:26.310549 29894 ts_tablet_manager.cc:1428] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2: Time spent starting tablet: real 0.100s user 0.013s sys 0.008s
I20260810 22:04:26.311502 29893 ts_tablet_manager.cc:1428] T dcfa54f1b7ef4b0b95226226bd78c40b P cfe74abd0282488f9a8ce6c152b761b2: Time spent starting tablet: real 0.108s user 0.035s sys 0.005s
I20260810 22:04:26.311806 29913 raft_consensus.cc:348] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:26.312665 29913 raft_consensus.cc:374] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:26.312968 29913 raft_consensus.cc:729] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfe74abd0282488f9a8ce6c152b761b2, State: Initialized, Role: FOLLOWER
I20260810 22:04:26.307269 29904 ts_tablet_manager.cc:1428] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed: Time spent starting tablet: real 0.134s user 0.017s sys 0.011s
I20260810 22:04:26.313746 29913 consensus_queue.cc:260] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:26.315764 29931 raft_consensus.cc:482] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:26.315976 29913 ts_tablet_manager.cc:1428] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2: Time spent starting tablet: real 0.073s user 0.004s sys 0.003s
I20260810 22:04:26.316370 29931 raft_consensus.cc:504] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } }
I20260810 22:04:26.318183 29940 leader_election.cc:288] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfe74abd0282488f9a8ce6c152b761b2 (127.27.17.131:38077), 6b67f5fd3c9943a7bf77411798547a55 (127.27.17.130:44819)
I20260810 22:04:26.323856 29931 leader_election.cc:288] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfe74abd0282488f9a8ce6c152b761b2 (127.27.17.131:38077), 21ef9e69018e484a8dbf082e54bbc07a (127.27.17.132:38435)
I20260810 22:04:26.327140 29912 tablet_bootstrap.cc:492] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed: No bootstrap required, opened a new log
I20260810 22:04:26.327579 29912 ts_tablet_manager.cc:1397] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed: Time spent bootstrapping tablet: real 0.127s user 0.007s sys 0.008s
I20260810 22:04:26.330148 29912 raft_consensus.cc:348] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } }
I20260810 22:04:26.330922 29912 raft_consensus.cc:374] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:26.331209 29912 raft_consensus.cc:729] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb15a3b74af456588e8733e2406f8ed, State: Initialized, Role: FOLLOWER
I20260810 22:04:26.331881 29912 consensus_queue.cc:260] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } }
I20260810 22:04:26.334250 29912 ts_tablet_manager.cc:1428] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260810 22:04:26.336006 29608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8690baf5d2f54129846bba669f5c6f3d" candidate_uuid: "9eb15a3b74af456588e8733e2406f8ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfe74abd0282488f9a8ce6c152b761b2" is_pre_election: true
I20260810 22:04:26.337020 29608 raft_consensus.cc:2454] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9eb15a3b74af456588e8733e2406f8ed in term 0.
I20260810 22:04:26.339903 29122 leader_election.cc:302] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9eb15a3b74af456588e8733e2406f8ed, cfe74abd0282488f9a8ce6c152b761b2; no voters:
I20260810 22:04:26.341054 29940 raft_consensus.cc:2781] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:26.341593 29940 raft_consensus.cc:482] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:26.341985 29940 raft_consensus.cc:3037] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:26.349241 29940 raft_consensus.cc:504] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:26.352061 29940 leader_election.cc:288] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [CANDIDATE]: Term 1 election: Requested vote from peers cfe74abd0282488f9a8ce6c152b761b2 (127.27.17.131:38077), 6b67f5fd3c9943a7bf77411798547a55 (127.27.17.130:44819)
I20260810 22:04:26.352463 29608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8690baf5d2f54129846bba669f5c6f3d" candidate_uuid: "9eb15a3b74af456588e8733e2406f8ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfe74abd0282488f9a8ce6c152b761b2"
I20260810 22:04:26.353042 29608 raft_consensus.cc:3037] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:26.355374 29403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8690baf5d2f54129846bba669f5c6f3d" candidate_uuid: "9eb15a3b74af456588e8733e2406f8ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b67f5fd3c9943a7bf77411798547a55" is_pre_election: true
I20260810 22:04:26.355791 29402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8690baf5d2f54129846bba669f5c6f3d" candidate_uuid: "9eb15a3b74af456588e8733e2406f8ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b67f5fd3c9943a7bf77411798547a55"
I20260810 22:04:26.356258 29403 raft_consensus.cc:2454] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9eb15a3b74af456588e8733e2406f8ed in term 0.
I20260810 22:04:26.359496 29608 raft_consensus.cc:2454] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9eb15a3b74af456588e8733e2406f8ed in term 1.
I20260810 22:04:26.360598 29122 leader_election.cc:302] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9eb15a3b74af456588e8733e2406f8ed, cfe74abd0282488f9a8ce6c152b761b2; no voters: 6b67f5fd3c9943a7bf77411798547a55
I20260810 22:04:26.361969 29940 raft_consensus.cc:2781] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:26.362471 29940 raft_consensus.cc:686] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [term 1 LEADER]: Becoming Leader. State: Replica: 9eb15a3b74af456588e8733e2406f8ed, State: Running, Role: LEADER
I20260810 22:04:26.363060 29608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff845c32f1f4b81b6b8d0052cf06968" candidate_uuid: "6b67f5fd3c9943a7bf77411798547a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfe74abd0282488f9a8ce6c152b761b2" is_pre_election: true
I20260810 22:04:26.363597 29608 raft_consensus.cc:2454] T fff845c32f1f4b81b6b8d0052cf06968 P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b67f5fd3c9943a7bf77411798547a55 in term 0.
I20260810 22:04:26.363557 29940 consensus_queue.cc:237] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:26.364265 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff845c32f1f4b81b6b8d0052cf06968" candidate_uuid: "6b67f5fd3c9943a7bf77411798547a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ef9e69018e484a8dbf082e54bbc07a" is_pre_election: true
I20260810 22:04:26.364900 29314 leader_election.cc:302] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b67f5fd3c9943a7bf77411798547a55, cfe74abd0282488f9a8ce6c152b761b2; no voters:
I20260810 22:04:26.365146 29800 raft_consensus.cc:2454] T fff845c32f1f4b81b6b8d0052cf06968 P 21ef9e69018e484a8dbf082e54bbc07a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b67f5fd3c9943a7bf77411798547a55 in term 0.
I20260810 22:04:26.365837 29931 raft_consensus.cc:2781] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:26.366195 29931 raft_consensus.cc:482] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:26.366518 29931 raft_consensus.cc:3037] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:26.369594 29929 raft_consensus.cc:482] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:26.369968 29929 raft_consensus.cc:504] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:26.371835 29929 leader_election.cc:288] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21ef9e69018e484a8dbf082e54bbc07a (127.27.17.132:38435), 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957)
I20260810 22:04:26.372598 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba3cc9bd6c9412dba778d21fd65181c" candidate_uuid: "6b67f5fd3c9943a7bf77411798547a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ef9e69018e484a8dbf082e54bbc07a" is_pre_election: true
I20260810 22:04:26.373214 29800 raft_consensus.cc:2454] T 7ba3cc9bd6c9412dba778d21fd65181c P 21ef9e69018e484a8dbf082e54bbc07a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b67f5fd3c9943a7bf77411798547a55 in term 0.
I20260810 22:04:26.374655 29316 leader_election.cc:302] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21ef9e69018e484a8dbf082e54bbc07a, 6b67f5fd3c9943a7bf77411798547a55; no voters:
I20260810 22:04:26.375422 29929 raft_consensus.cc:2781] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:26.375801 29929 raft_consensus.cc:482] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:26.374676 29931 raft_consensus.cc:504] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } }
I20260810 22:04:26.376122 29929 raft_consensus.cc:3037] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:26.377646 29931 leader_election.cc:288] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 [CANDIDATE]: Term 1 election: Requested vote from peers cfe74abd0282488f9a8ce6c152b761b2 (127.27.17.131:38077), 21ef9e69018e484a8dbf082e54bbc07a (127.27.17.132:38435)
I20260810 22:04:26.378638 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff845c32f1f4b81b6b8d0052cf06968" candidate_uuid: "6b67f5fd3c9943a7bf77411798547a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ef9e69018e484a8dbf082e54bbc07a"
I20260810 22:04:26.379187 29800 raft_consensus.cc:3037] T fff845c32f1f4b81b6b8d0052cf06968 P 21ef9e69018e484a8dbf082e54bbc07a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:26.380049 29608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff845c32f1f4b81b6b8d0052cf06968" candidate_uuid: "6b67f5fd3c9943a7bf77411798547a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfe74abd0282488f9a8ce6c152b761b2"
I20260810 22:04:26.379303 29015 catalog_manager.cc:5495] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129). New cstate: current_term: 1 leader_uuid: "9eb15a3b74af456588e8733e2406f8ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } health_report { overall_health: UNKNOWN } } }
I20260810 22:04:26.380791 29608 raft_consensus.cc:3037] T fff845c32f1f4b81b6b8d0052cf06968 P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:26.382633 29929 raft_consensus.cc:504] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:26.383780 29940 raft_consensus.cc:482] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:26.384218 29940 raft_consensus.cc:504] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } }
I20260810 22:04:26.386308 29940 leader_election.cc:288] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfe74abd0282488f9a8ce6c152b761b2 (127.27.17.131:38077), 21ef9e69018e484a8dbf082e54bbc07a (127.27.17.132:38435)
I20260810 22:04:26.388154 29607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabe053775754da49c6bc46e79d13bf3" candidate_uuid: "9eb15a3b74af456588e8733e2406f8ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfe74abd0282488f9a8ce6c152b761b2" is_pre_election: true
I20260810 22:04:26.388935 29607 raft_consensus.cc:2454] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9eb15a3b74af456588e8733e2406f8ed in term 0.
I20260810 22:04:26.389642 29800 raft_consensus.cc:2454] T fff845c32f1f4b81b6b8d0052cf06968 P 21ef9e69018e484a8dbf082e54bbc07a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b67f5fd3c9943a7bf77411798547a55 in term 1.
I20260810 22:04:26.390159 29122 leader_election.cc:302] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9eb15a3b74af456588e8733e2406f8ed, cfe74abd0282488f9a8ce6c152b761b2; no voters:
I20260810 22:04:26.390901 29316 leader_election.cc:302] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21ef9e69018e484a8dbf082e54bbc07a, 6b67f5fd3c9943a7bf77411798547a55; no voters:
I20260810 22:04:26.391273 29940 raft_consensus.cc:2781] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:26.391634 29940 raft_consensus.cc:482] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:26.391726 29931 raft_consensus.cc:2781] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:26.391957 29940 raft_consensus.cc:3037] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:26.393231 29931 raft_consensus.cc:686] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 [term 1 LEADER]: Becoming Leader. State: Replica: 6b67f5fd3c9943a7bf77411798547a55, State: Running, Role: LEADER
I20260810 22:04:26.393414 29608 raft_consensus.cc:2454] T fff845c32f1f4b81b6b8d0052cf06968 P cfe74abd0282488f9a8ce6c152b761b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b67f5fd3c9943a7bf77411798547a55 in term 1.
I20260810 22:04:26.394385 29931 consensus_queue.cc:237] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } }
W20260810 22:04:26.396014 29476 tablet.cc:2369] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:04:26.396265 29929 leader_election.cc:288] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 [CANDIDATE]: Term 1 election: Requested vote from peers 21ef9e69018e484a8dbf082e54bbc07a (127.27.17.132:38435), 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957)
I20260810 22:04:26.399494 29799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba3cc9bd6c9412dba778d21fd65181c" candidate_uuid: "6b67f5fd3c9943a7bf77411798547a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ef9e69018e484a8dbf082e54bbc07a"
I20260810 22:04:26.400220 29799 raft_consensus.cc:3037] T 7ba3cc9bd6c9412dba778d21fd65181c P 21ef9e69018e484a8dbf082e54bbc07a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:26.400312 29217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba3cc9bd6c9412dba778d21fd65181c" candidate_uuid: "6b67f5fd3c9943a7bf77411798547a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eb15a3b74af456588e8733e2406f8ed" is_pre_election: true
I20260810 22:04:26.401414 29217 raft_consensus.cc:2454] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b67f5fd3c9943a7bf77411798547a55 in term 0.
I20260810 22:04:26.403260 29216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba3cc9bd6c9412dba778d21fd65181c" candidate_uuid: "6b67f5fd3c9943a7bf77411798547a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eb15a3b74af456588e8733e2406f8ed"
I20260810 22:04:26.404109 29216 raft_consensus.cc:3037] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:26.404130 29798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabe053775754da49c6bc46e79d13bf3" candidate_uuid: "9eb15a3b74af456588e8733e2406f8ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ef9e69018e484a8dbf082e54bbc07a" is_pre_election: true
I20260810 22:04:26.404888 29798 raft_consensus.cc:2454] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9eb15a3b74af456588e8733e2406f8ed in term 0.
I20260810 22:04:26.407701 29799 raft_consensus.cc:2454] T 7ba3cc9bd6c9412dba778d21fd65181c P 21ef9e69018e484a8dbf082e54bbc07a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b67f5fd3c9943a7bf77411798547a55 in term 1.
I20260810 22:04:26.408718 29316 leader_election.cc:302] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21ef9e69018e484a8dbf082e54bbc07a, 6b67f5fd3c9943a7bf77411798547a55; no voters:
I20260810 22:04:26.409651 29929 raft_consensus.cc:2781] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:26.410120 29929 raft_consensus.cc:686] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 [term 1 LEADER]: Becoming Leader. State: Replica: 6b67f5fd3c9943a7bf77411798547a55, State: Running, Role: LEADER
I20260810 22:04:26.409686 29940 raft_consensus.cc:504] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } }
I20260810 22:04:26.410842 29929 consensus_queue.cc:237] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:26.411844 29940 leader_election.cc:288] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed [CANDIDATE]: Term 1 election: Requested vote from peers cfe74abd0282488f9a8ce6c152b761b2 (127.27.17.131:38077), 21ef9e69018e484a8dbf082e54bbc07a (127.27.17.132:38435)
I20260810 22:04:26.412659 29608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabe053775754da49c6bc46e79d13bf3" candidate_uuid: "9eb15a3b74af456588e8733e2406f8ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfe74abd0282488f9a8ce6c152b761b2"
I20260810 22:04:26.413126 29216 raft_consensus.cc:2454] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b67f5fd3c9943a7bf77411798547a55 in term 1.
I20260810 22:04:26.413327 29608 raft_consensus.cc:3037] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:26.413725 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabe053775754da49c6bc46e79d13bf3" candidate_uuid: "9eb15a3b74af456588e8733e2406f8ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ef9e69018e484a8dbf082e54bbc07a"
I20260810 22:04:26.414291 29800 raft_consensus.cc:3037] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:26.420050 29608 raft_consensus.cc:2454] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9eb15a3b74af456588e8733e2406f8ed in term 1.
I20260810 22:04:26.422513 29122 leader_election.cc:302] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9eb15a3b74af456588e8733e2406f8ed, cfe74abd0282488f9a8ce6c152b761b2; no voters:
I20260810 22:04:26.423517 29940 raft_consensus.cc:2781] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:26.424129 29940 raft_consensus.cc:686] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed [term 1 LEADER]: Becoming Leader. State: Replica: 9eb15a3b74af456588e8733e2406f8ed, State: Running, Role: LEADER
I20260810 22:04:26.425076 29940 consensus_queue.cc:237] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } }
I20260810 22:04:26.429533 29800 raft_consensus.cc:2454] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9eb15a3b74af456588e8733e2406f8ed in term 1.
W20260810 22:04:26.438849 29281 tablet.cc:2369] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:04:26.440691 29929 raft_consensus.cc:482] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:26.441576 29929 raft_consensus.cc:504] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } }
I20260810 22:04:26.445266 29929 leader_election.cc:288] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957), cfe74abd0282488f9a8ce6c152b761b2 (127.27.17.131:38077)
I20260810 22:04:26.446343 29216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcfa54f1b7ef4b0b95226226bd78c40b" candidate_uuid: "6b67f5fd3c9943a7bf77411798547a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eb15a3b74af456588e8733e2406f8ed" is_pre_election: true
I20260810 22:04:26.446911 29216 raft_consensus.cc:2454] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b67f5fd3c9943a7bf77411798547a55 in term 0.
I20260810 22:04:26.446630 29608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcfa54f1b7ef4b0b95226226bd78c40b" candidate_uuid: "6b67f5fd3c9943a7bf77411798547a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfe74abd0282488f9a8ce6c152b761b2" is_pre_election: true
I20260810 22:04:26.447435 29608 raft_consensus.cc:2454] T dcfa54f1b7ef4b0b95226226bd78c40b P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b67f5fd3c9943a7bf77411798547a55 in term 0.
I20260810 22:04:26.448019 29315 leader_election.cc:302] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b67f5fd3c9943a7bf77411798547a55, 9eb15a3b74af456588e8733e2406f8ed; no voters:
I20260810 22:04:26.449245 29929 raft_consensus.cc:2781] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:26.449672 29929 raft_consensus.cc:482] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:26.449965 29929 raft_consensus.cc:3037] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:26.454779 29929 raft_consensus.cc:504] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } }
I20260810 22:04:26.458026 29216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcfa54f1b7ef4b0b95226226bd78c40b" candidate_uuid: "6b67f5fd3c9943a7bf77411798547a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eb15a3b74af456588e8733e2406f8ed"
I20260810 22:04:26.458694 29216 raft_consensus.cc:3037] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:26.459206 29014 catalog_manager.cc:5495] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b67f5fd3c9943a7bf77411798547a55 (127.27.17.130). New cstate: current_term: 1 leader_uuid: "6b67f5fd3c9943a7bf77411798547a55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } health_report { overall_health: UNKNOWN } } }
I20260810 22:04:26.462785 29929 leader_election.cc:288] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [CANDIDATE]: Term 1 election: Requested vote from peers 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957), cfe74abd0282488f9a8ce6c152b761b2 (127.27.17.131:38077)
I20260810 22:04:26.465555 29216 raft_consensus.cc:2454] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b67f5fd3c9943a7bf77411798547a55 in term 1.
I20260810 22:04:26.464663 29015 catalog_manager.cc:5495] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129). New cstate: current_term: 1 leader_uuid: "9eb15a3b74af456588e8733e2406f8ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } health_report { overall_health: HEALTHY } } }
I20260810 22:04:26.466792 29315 leader_election.cc:302] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b67f5fd3c9943a7bf77411798547a55, 9eb15a3b74af456588e8733e2406f8ed; no voters:
I20260810 22:04:26.467919 29929 raft_consensus.cc:2781] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:26.464114 29608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcfa54f1b7ef4b0b95226226bd78c40b" candidate_uuid: "6b67f5fd3c9943a7bf77411798547a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfe74abd0282488f9a8ce6c152b761b2"
I20260810 22:04:26.468401 29929 raft_consensus.cc:686] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [term 1 LEADER]: Becoming Leader. State: Replica: 6b67f5fd3c9943a7bf77411798547a55, State: Running, Role: LEADER
I20260810 22:04:26.468497 29608 raft_consensus.cc:3037] T dcfa54f1b7ef4b0b95226226bd78c40b P cfe74abd0282488f9a8ce6c152b761b2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:26.469252 29929 consensus_queue.cc:237] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } }
I20260810 22:04:26.473045 29608 raft_consensus.cc:2454] T dcfa54f1b7ef4b0b95226226bd78c40b P cfe74abd0282488f9a8ce6c152b761b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b67f5fd3c9943a7bf77411798547a55 in term 1.
I20260810 22:04:26.482376 29015 catalog_manager.cc:5495] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b67f5fd3c9943a7bf77411798547a55 (127.27.17.130). New cstate: current_term: 1 leader_uuid: "6b67f5fd3c9943a7bf77411798547a55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } health_report { overall_health: HEALTHY } } }
I20260810 22:04:26.483722 29015 catalog_manager.cc:5495] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b67f5fd3c9943a7bf77411798547a55 (127.27.17.130). New cstate: current_term: 1 leader_uuid: "6b67f5fd3c9943a7bf77411798547a55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } health_report { overall_health: UNKNOWN } } }
W20260810 22:04:26.495666 29675 tablet.cc:2369] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:04:26.635344 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 29088
I20260810 22:04:26.675581 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.129:37957
--local_ip_for_outbound_sockets=127.27.17.129
--tserver_master_addrs=127.27.17.190:35505
--webserver_port=38101
--webserver_interface=127.27.17.129
--env_inject_eio_globs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-1/**
--env_inject_eio=1.0
--crash_on_eio=false
--builtin_ntp_servers=127.27.17.148:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260810 22:04:26.855268 29929 consensus_queue.cc:1035] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 22:04:26.902613 29929 consensus_queue.cc:1035] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:04:26.913282 29932 consensus_queue.cc:1035] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 22:04:26.928051 29931 consensus_queue.cc:1035] T fff845c32f1f4b81b6b8d0052cf06968 P 6b67f5fd3c9943a7bf77411798547a55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 22:04:26.939055 29315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.129:37957: connect: Connection refused (error 111)
W20260810 22:04:26.952237 29315 consensus_peers.cc:487] T 7ba3cc9bd6c9412dba778d21fd65181c P 6b67f5fd3c9943a7bf77411798547a55 -> Peer 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957): Couldn't send request to peer 9eb15a3b74af456588e8733e2406f8ed. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:37957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 22:04:26.978312 29315 consensus_peers.cc:487] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 -> Peer 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957): Couldn't send request to peer 9eb15a3b74af456588e8733e2406f8ed. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:37957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 22:04:27.093181 29965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:27.093866 29965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:27.094563 29965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:27.125886 29965 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20260810 22:04:27.126345 29965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:27.127030 29965 flags.cc:425] Enabled experimental flag: --crash_on_eio=false
W20260810 22:04:27.127576 29965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.129
I20260810 22:04:27.164007 29965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.129:37957
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.27.17.129
--webserver_port=38101
--tserver_master_addrs=127.27.17.190:35505
--env_inject_eio=1
--env_inject_eio_globs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-1/**
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.129
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:27.165405 29965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260810 22:04:27.166272 29965 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260810 22:04:27.166452 29965 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260810 22:04:27.167420 29965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:27.186028 29989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:27.878990 29932 raft_consensus.cc:482] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:27.879473 29932 raft_consensus.cc:504] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:27.881410 29932 leader_election.cc:288] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfe74abd0282488f9a8ce6c152b761b2 (127.27.17.131:38077), 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957)
I20260810 22:04:27.883188 29608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8690baf5d2f54129846bba669f5c6f3d" candidate_uuid: "6b67f5fd3c9943a7bf77411798547a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfe74abd0282488f9a8ce6c152b761b2" is_pre_election: true
I20260810 22:04:27.884003 29608 raft_consensus.cc:2379] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b67f5fd3c9943a7bf77411798547a55 in current term 1: Already voted for candidate 9eb15a3b74af456588e8733e2406f8ed in this term.
W20260810 22:04:27.889551 29315 leader_election.cc:334] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957): Network error: Client connection negotiation failed: client connection to 127.27.17.129:37957: connect: Connection refused (error 111)
I20260810 22:04:27.890085 29315 leader_election.cc:302] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b67f5fd3c9943a7bf77411798547a55; no voters: 9eb15a3b74af456588e8733e2406f8ed, cfe74abd0282488f9a8ce6c152b761b2
I20260810 22:04:27.890877 29932 raft_consensus.cc:3037] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:27.902768 29932 raft_consensus.cc:2726] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 22:04:28.028820 29996 raft_consensus.cc:482] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:28.029217 29996 raft_consensus.cc:504] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } }
I20260810 22:04:28.031426 29996 leader_election.cc:288] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cfe74abd0282488f9a8ce6c152b761b2 (127.27.17.131:38077), 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957)
W20260810 22:04:28.049296 29714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.129:37957: connect: Connection refused (error 111)
W20260810 22:04:28.053215 29714 leader_election.cc:334] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957): Network error: Client connection negotiation failed: client connection to 127.27.17.129:37957: connect: Connection refused (error 111)
I20260810 22:04:28.061825 29608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabe053775754da49c6bc46e79d13bf3" candidate_uuid: "21ef9e69018e484a8dbf082e54bbc07a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfe74abd0282488f9a8ce6c152b761b2" is_pre_election: true
I20260810 22:04:28.062774 29608 raft_consensus.cc:2454] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21ef9e69018e484a8dbf082e54bbc07a in term 1.
I20260810 22:04:28.064229 29713 leader_election.cc:302] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21ef9e69018e484a8dbf082e54bbc07a, cfe74abd0282488f9a8ce6c152b761b2; no voters: 9eb15a3b74af456588e8733e2406f8ed
I20260810 22:04:28.065428 29996 raft_consensus.cc:2781] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 22:04:28.065727 29996 raft_consensus.cc:482] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:28.065994 29996 raft_consensus.cc:3037] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:04:28.070343 29996 raft_consensus.cc:504] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } }
I20260810 22:04:28.071784 29996 leader_election.cc:288] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [CANDIDATE]: Term 2 election: Requested vote from peers cfe74abd0282488f9a8ce6c152b761b2 (127.27.17.131:38077), 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957)
I20260810 22:04:28.073050 29608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabe053775754da49c6bc46e79d13bf3" candidate_uuid: "21ef9e69018e484a8dbf082e54bbc07a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfe74abd0282488f9a8ce6c152b761b2"
I20260810 22:04:28.073655 29608 raft_consensus.cc:3037] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2 [term 1 FOLLOWER]: Advancing to term 2
W20260810 22:04:28.075722 29714 leader_election.cc:334] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957): Network error: Client connection negotiation failed: client connection to 127.27.17.129:37957: connect: Connection refused (error 111)
I20260810 22:04:28.078265 29608 raft_consensus.cc:2454] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21ef9e69018e484a8dbf082e54bbc07a in term 2.
I20260810 22:04:28.079113 29713 leader_election.cc:302] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21ef9e69018e484a8dbf082e54bbc07a, cfe74abd0282488f9a8ce6c152b761b2; no voters: 9eb15a3b74af456588e8733e2406f8ed
I20260810 22:04:28.079771 29996 raft_consensus.cc:2781] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:04:28.081847 29996 raft_consensus.cc:686] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [term 2 LEADER]: Becoming Leader. State: Replica: 21ef9e69018e484a8dbf082e54bbc07a, State: Running, Role: LEADER
I20260810 22:04:28.082741 29996 consensus_queue.cc:237] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } }
I20260810 22:04:28.097481 29015 catalog_manager.cc:5495] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a reported cstate change: term changed from 1 to 2, leader changed from 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129) to 21ef9e69018e484a8dbf082e54bbc07a (127.27.17.132). New cstate: current_term: 2 leader_uuid: "21ef9e69018e484a8dbf082e54bbc07a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } health_report { overall_health: UNKNOWN } } }
I20260810 22:04:27.187577 29965 server_base.cc:1034] running on GCE node
W20260810 22:04:27.187296 29992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:27.187288 29990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:28.416603 29965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:28.419711 29965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:28.421168 29965 hybrid_clock.cc:648] HybridClock initialized: now 1786399468421125 us; error 45 us; skew 500 ppm
I20260810 22:04:28.422062 29965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:28.429587 29965 webserver.cc:460] Webserver started at http://127.27.17.129:38101/ using document root <none> and password file <none>
I20260810 22:04:28.431330 29965 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:28.431594 29965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
E20260810 22:04:28.434603 29965 env_posix.cc:432] I/O error, context: INJECTED FAILURE
E20260810 22:04:28.439965 29965 env_posix.cc:432] I/O error, context: INJECTED FAILURE
W20260810 22:04:28.452661 29965 data_dirs.cc:318] instance /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-1/data/block_manager_instance has unknown UUID 8321b12bfa2e4680827beeea653f7f14
I20260810 22:04:28.453859 29965 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.005s sys 0.010s
I20260810 22:04:28.456112 29965 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:28.456354 29965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/wal
uuid: "9eb15a3b74af456588e8733e2406f8ed"
format_stamp: "Formatted at 2026-08-10 22:04:19 on dist-test-slave-bw88"
I20260810 22:04:28.458950 29965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/wal
4 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260810 22:04:28.504081 29965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:28.505807 29965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:28.506449 29965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:28.509441 29965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:04:28.516520 30027 ts_tablet_manager.cc:542] Loading tablet metadata (0/4 complete)
I20260810 22:04:28.534090 29965 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20260810 22:04:28.534401 29965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.020s user 0.005s sys 0.007s
I20260810 22:04:28.534713 29965 ts_tablet_manager.cc:594] Registering tablets (0/4 complete)
I20260810 22:04:28.537681 30032 raft_consensus.cc:482] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:28.538090 30032 raft_consensus.cc:504] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:28.542783 30031 tablet_bootstrap.cc:492] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed: Bootstrap starting.
W20260810 22:04:28.544277 29714 consensus_peers.cc:487] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a -> Peer 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957): Couldn't send request to peer 9eb15a3b74af456588e8733e2406f8ed. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:37957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 22:04:28.546988 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.129:37957: connect: Connection refused (error 111)
I20260810 22:04:28.550675 30032 leader_election.cc:288] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957), 6b67f5fd3c9943a7bf77411798547a55 (127.27.17.130:44819)
I20260810 22:04:28.551057 30028 tablet_bootstrap.cc:492] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed: Bootstrap starting.
I20260810 22:04:28.559171 30029 tablet_bootstrap.cc:492] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed: Bootstrap starting.
E20260810 22:04:28.559894 30029 ts_tablet_manager.cc:1419] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory
I20260810 22:04:28.560262 30029 ts_tablet_manager.cc:1397] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:04:28.560595 30029 tablet_replica.cc:330] stopping tablet replica
W20260810 22:04:28.565794 29511 leader_election.cc:334] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957): Network error: Client connection negotiation failed: client connection to 127.27.17.129:37957: connect: Connection refused (error 111)
I20260810 22:04:28.567350 29965 ts_tablet_manager.cc:610] Registered 4 tablets
I20260810 22:04:28.567632 29965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.033s user 0.025s sys 0.006s
I20260810 22:04:28.567659 30027 tablet_bootstrap.cc:492] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed: Bootstrap starting.
E20260810 22:04:28.568385 30027 ts_tablet_manager.cc:1419] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory
I20260810 22:04:28.568722 30027 ts_tablet_manager.cc:1397] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:04:28.568998 30027 tablet_replica.cc:330] stopping tablet replica
I20260810 22:04:28.576792 29402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8690baf5d2f54129846bba669f5c6f3d" candidate_uuid: "cfe74abd0282488f9a8ce6c152b761b2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b67f5fd3c9943a7bf77411798547a55" is_pre_election: true
I20260810 22:04:28.577607 29402 raft_consensus.cc:2454] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfe74abd0282488f9a8ce6c152b761b2 in term 1.
I20260810 22:04:28.579030 29508 leader_election.cc:302] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b67f5fd3c9943a7bf77411798547a55, cfe74abd0282488f9a8ce6c152b761b2; no voters: 9eb15a3b74af456588e8733e2406f8ed
I20260810 22:04:28.580255 30032 raft_consensus.cc:2781] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 22:04:28.580577 30032 raft_consensus.cc:482] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:28.580883 30032 raft_consensus.cc:3037] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:04:28.569267 30027 raft_consensus.cc:2229] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:04:28.581419 30027 raft_consensus.cc:2258] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:04:28.561002 30029 raft_consensus.cc:2229] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:04:28.582322 30029 raft_consensus.cc:2258] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:04:28.583892 29608 raft_consensus.cc:1261] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2 [term 2 FOLLOWER]: Refusing update from remote peer 21ef9e69018e484a8dbf082e54bbc07a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260810 22:04:28.585657 30003 consensus_queue.cc:1035] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 22:04:28.591846 30032 raft_consensus.cc:504] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
W20260810 22:04:28.602180 29511 leader_election.cc:334] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957): Network error: Client connection negotiation failed: client connection to 127.27.17.129:37957: connect: Connection refused (error 111)
I20260810 22:04:28.605701 29402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8690baf5d2f54129846bba669f5c6f3d" candidate_uuid: "cfe74abd0282488f9a8ce6c152b761b2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b67f5fd3c9943a7bf77411798547a55"
I20260810 22:04:28.606381 29402 raft_consensus.cc:3037] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:04:28.620419 29402 raft_consensus.cc:2454] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfe74abd0282488f9a8ce6c152b761b2 in term 2.
I20260810 22:04:28.621464 30032 leader_election.cc:288] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [CANDIDATE]: Term 2 election: Requested vote from peers 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957), 6b67f5fd3c9943a7bf77411798547a55 (127.27.17.130:44819)
I20260810 22:04:28.623020 29508 leader_election.cc:302] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b67f5fd3c9943a7bf77411798547a55, cfe74abd0282488f9a8ce6c152b761b2; no voters: 9eb15a3b74af456588e8733e2406f8ed
I20260810 22:04:28.631014 30032 raft_consensus.cc:2781] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:04:28.633263 30032 raft_consensus.cc:686] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [term 2 LEADER]: Becoming Leader. State: Replica: cfe74abd0282488f9a8ce6c152b761b2, State: Running, Role: LEADER
I20260810 22:04:28.634569 30028 log.cc:824] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:28.636477 30032 consensus_queue.cc:237] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:28.652616 30028 tablet_bootstrap.cc:492] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260810 22:04:28.654420 30028 tablet_bootstrap.cc:492] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed: Bootstrap complete.
I20260810 22:04:28.657182 30028 ts_tablet_manager.cc:1397] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed: Time spent bootstrapping tablet: real 0.107s user 0.032s sys 0.058s
I20260810 22:04:28.665777 29015 catalog_manager.cc:5495] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 reported cstate change: term changed from 1 to 2, leader changed from 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129) to cfe74abd0282488f9a8ce6c152b761b2 (127.27.17.131). New cstate: current_term: 2 leader_uuid: "cfe74abd0282488f9a8ce6c152b761b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } health_report { overall_health: UNKNOWN } } }
I20260810 22:04:28.674984 30031 tablet_bootstrap.cc:492] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:04:28.676399 30031 tablet_bootstrap.cc:492] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed: Bootstrap complete.
I20260810 22:04:28.678779 30031 ts_tablet_manager.cc:1397] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed: Time spent bootstrapping tablet: real 0.136s user 0.047s sys 0.039s
I20260810 22:04:28.708137 30028 raft_consensus.cc:348] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:28.709070 30031 raft_consensus.cc:348] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:28.709865 30031 raft_consensus.cc:729] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb15a3b74af456588e8733e2406f8ed, State: Initialized, Role: FOLLOWER
I20260810 22:04:28.710208 30028 raft_consensus.cc:729] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb15a3b74af456588e8733e2406f8ed, State: Initialized, Role: FOLLOWER
I20260810 22:04:28.711156 30028 consensus_queue.cc:260] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:28.718811 30031 consensus_queue.cc:260] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } }
I20260810 22:04:28.724197 30028 ts_tablet_manager.cc:1428] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed: Time spent starting tablet: real 0.066s user 0.042s sys 0.018s
I20260810 22:04:28.728288 30031 ts_tablet_manager.cc:1428] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed: Time spent starting tablet: real 0.035s user 0.002s sys 0.011s
I20260810 22:04:28.860693 29965 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.129:37957
I20260810 22:04:28.861274 30200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.129:37957 every 8 connection(s)
I20260810 22:04:28.864550 29965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
I20260810 22:04:28.867111 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 29965
W20260810 22:04:28.894177 30203 tablet.cc:2369] T 7ba3cc9bd6c9412dba778d21fd65181c P 9eb15a3b74af456588e8733e2406f8ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:04:28.919332 30201 heartbeater.cc:347] Connected to a master server at 127.27.17.190:35505
I20260810 22:04:28.919857 30201 heartbeater.cc:464] Registering TS with master...
I20260810 22:04:28.921137 30201 heartbeater.cc:511] Master 127.27.17.190:35505 requested a full tablet report, sending...
I20260810 22:04:28.926457 29015 ts_manager.cc:194] Re-registered known tserver with Master: 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957)
W20260810 22:04:28.929085 29015 catalog_manager.cc:5408] Tablet dcfa54f1b7ef4b0b95226226bd78c40b (table test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]) has failed on TS 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957): IO error: some tablet data is in a failed directory
W20260810 22:04:28.929687 29015 catalog_manager.cc:5408] Tablet cabe053775754da49c6bc46e79d13bf3 (table test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]) has failed on TS 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957): IO error: some tablet data is in a failed directory
I20260810 22:04:28.932874 29015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.129:53975
I20260810 22:04:29.048924 29402 raft_consensus.cc:1261] T 8690baf5d2f54129846bba669f5c6f3d P 6b67f5fd3c9943a7bf77411798547a55 [term 2 FOLLOWER]: Refusing update from remote peer cfe74abd0282488f9a8ce6c152b761b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260810 22:04:29.050458 30032 consensus_queue.cc:1035] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:04:29.080055 30140 raft_consensus.cc:3037] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:04:29.090862 30140 raft_consensus.cc:1261] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [term 2 FOLLOWER]: Refusing update from remote peer cfe74abd0282488f9a8ce6c152b761b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260810 22:04:29.091224 30140 pending_rounds.cc:76] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed: Aborting all ops after (but not including) 0
I20260810 22:04:29.091435 30140 pending_rounds.cc:97] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260810 22:04:29.091715 30140 raft_consensus.cc:2866] T 8690baf5d2f54129846bba669f5c6f3d P 9eb15a3b74af456588e8733e2406f8ed [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 22:04:29.093804 30032 consensus_queue.cc:1035] T 8690baf5d2f54129846bba669f5c6f3d P cfe74abd0282488f9a8ce6c152b761b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9eb15a3b74af456588e8733e2406f8ed" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:04:29.102104 30201 heartbeater.cc:503] Master 127.27.17.190:35505 was elected leader, sending a full tablet report...
I20260810 22:04:29.120361 29315 consensus_queue.cc:579] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [LEADER]: The tablet replica hosted on peer 9eb15a3b74af456588e8733e2406f8ed has failed
W20260810 22:04:29.120522 29015 catalog_manager.cc:5408] Tablet dcfa54f1b7ef4b0b95226226bd78c40b (table test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]) has failed on TS 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957): IO error: some tablet data is in a failed directory
W20260810 22:04:29.121148 29015 catalog_manager.cc:5408] Tablet cabe053775754da49c6bc46e79d13bf3 (table test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]) has failed on TS 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957): IO error: some tablet data is in a failed directory
I20260810 22:04:29.130066 29402 consensus_queue.cc:237] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } }
I20260810 22:04:29.134600 29607 raft_consensus.cc:1261] T dcfa54f1b7ef4b0b95226226bd78c40b P cfe74abd0282488f9a8ce6c152b761b2 [term 1 FOLLOWER]: Refusing update from remote peer 6b67f5fd3c9943a7bf77411798547a55: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:04:29.135789 29932 consensus_queue.cc:1035] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 22:04:29.142796 30227 raft_consensus.cc:2932] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } }
I20260810 22:04:29.144625 29607 raft_consensus.cc:2932] T dcfa54f1b7ef4b0b95226226bd78c40b P cfe74abd0282488f9a8ce6c152b761b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } }
I20260810 22:04:29.153648 28993 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet dcfa54f1b7ef4b0b95226226bd78c40b with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 22:04:29.156904 29015 catalog_manager.cc:5495] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 reported cstate change: config changed from index -1 to 2, VOTER 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129) evicted. New cstate: current_term: 1 leader_uuid: "6b67f5fd3c9943a7bf77411798547a55" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } health_report { overall_health: HEALTHY } } }
I20260810 22:04:29.171962 29402 consensus_queue.cc:237] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: NON_VOTER last_known_addr { host: "127.27.17.132" port: 38435 } attrs { promote: true } }
W20260810 22:04:29.179317 29316 consensus_peers.cc:487] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 -> Peer 21ef9e69018e484a8dbf082e54bbc07a (127.27.17.132:38435): Couldn't send request to peer 21ef9e69018e484a8dbf082e54bbc07a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcfa54f1b7ef4b0b95226226bd78c40b. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:29.180519 29714 consensus_queue.cc:579] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [LEADER]: The tablet replica hosted on peer 9eb15a3b74af456588e8733e2406f8ed has failed
I20260810 22:04:29.182811 29607 raft_consensus.cc:1261] T dcfa54f1b7ef4b0b95226226bd78c40b P cfe74abd0282488f9a8ce6c152b761b2 [term 1 FOLLOWER]: Refusing update from remote peer 6b67f5fd3c9943a7bf77411798547a55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 22:04:29.184880 30227 consensus_queue.cc:1035] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 22:04:29.191107 29800 consensus_queue.cc:237] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } }
I20260810 22:04:29.194861 29932 raft_consensus.cc:2932] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 21ef9e69018e484a8dbf082e54bbc07a (127.27.17.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: NON_VOTER last_known_addr { host: "127.27.17.132" port: 38435 } attrs { promote: true } } }
I20260810 22:04:29.197244 29607 raft_consensus.cc:1261] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2 [term 2 FOLLOWER]: Refusing update from remote peer 21ef9e69018e484a8dbf082e54bbc07a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 22:04:29.198771 29996 consensus_queue.cc:1035] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 22:04:29.199467 29607 raft_consensus.cc:2932] T dcfa54f1b7ef4b0b95226226bd78c40b P cfe74abd0282488f9a8ce6c152b761b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 21ef9e69018e484a8dbf082e54bbc07a (127.27.17.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: NON_VOTER last_known_addr { host: "127.27.17.132" port: 38435 } attrs { promote: true } } }
I20260810 22:04:29.205293 28993 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dcfa54f1b7ef4b0b95226226bd78c40b with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260810 22:04:29.207430 30003 raft_consensus.cc:2932] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } }
I20260810 22:04:29.212498 29013 catalog_manager.cc:5495] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 reported cstate change: config changed from index 2 to 3, NON_VOTER 21ef9e69018e484a8dbf082e54bbc07a (127.27.17.132) added. New cstate: current_term: 1 leader_uuid: "6b67f5fd3c9943a7bf77411798547a55" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: NON_VOTER last_known_addr { host: "127.27.17.132" port: 38435 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260810 22:04:29.213605 29607 raft_consensus.cc:2932] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } }
I20260810 22:04:29.231599 29000 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet cabe053775754da49c6bc46e79d13bf3 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 22:04:29.231283 29015 catalog_manager.cc:5495] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2 reported cstate change: config changed from index -1 to 2, VOTER 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129) evicted. New cstate: current_term: 2 leader_uuid: "21ef9e69018e484a8dbf082e54bbc07a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } }
I20260810 22:04:29.246990 30118 tablet_service.cc:1514] Processing DeleteTablet for tablet dcfa54f1b7ef4b0b95226226bd78c40b with delete_type TABLET_DATA_TOMBSTONED (TS 9eb15a3b74af456588e8733e2406f8ed not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:49900
I20260810 22:04:29.251366 30250 ts_tablet_manager.cc:1905] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 22:04:29.251451 30117 tablet_service.cc:1514] Processing DeleteTablet for tablet cabe053775754da49c6bc46e79d13bf3 with delete_type TABLET_DATA_TOMBSTONED (TS 9eb15a3b74af456588e8733e2406f8ed not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:49900
I20260810 22:04:29.267406 29800 consensus_queue.cc:237] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: NON_VOTER last_known_addr { host: "127.27.17.130" port: 44819 } attrs { promote: true } }
I20260810 22:04:29.270371 30252 ts_tablet_manager.cc:1905] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 22:04:29.276116 29607 raft_consensus.cc:1261] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2 [term 2 FOLLOWER]: Refusing update from remote peer 21ef9e69018e484a8dbf082e54bbc07a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260810 22:04:29.277724 30003 consensus_queue.cc:1035] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 22:04:29.278081 30250 ts_tablet_manager.cc:1918] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260810 22:04:29.278451 30250 log.cc:1196] T dcfa54f1b7ef4b0b95226226bd78c40b P 9eb15a3b74af456588e8733e2406f8ed: Deleting WAL directory at /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/wal/wals/dcfa54f1b7ef4b0b95226226bd78c40b
I20260810 22:04:29.280683 28998 catalog_manager.cc:4841] TS 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957): tablet dcfa54f1b7ef4b0b95226226bd78c40b (table test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]) successfully deleted
I20260810 22:04:29.294051 30241 raft_consensus.cc:2932] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, NON_VOTER 6b67f5fd3c9943a7bf77411798547a55 (127.27.17.130) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: NON_VOTER last_known_addr { host: "127.27.17.130" port: 44819 } attrs { promote: true } } }
I20260810 22:04:29.295586 29607 raft_consensus.cc:2932] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, NON_VOTER 6b67f5fd3c9943a7bf77411798547a55 (127.27.17.130) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: NON_VOTER last_known_addr { host: "127.27.17.130" port: 44819 } attrs { promote: true } } }
I20260810 22:04:29.298911 30252 ts_tablet_manager.cc:1918] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260810 22:04:29.299266 30252 log.cc:1196] T cabe053775754da49c6bc46e79d13bf3 P 9eb15a3b74af456588e8733e2406f8ed: Deleting WAL directory at /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/wal/wals/cabe053775754da49c6bc46e79d13bf3
I20260810 22:04:29.301433 28998 catalog_manager.cc:4841] TS 9eb15a3b74af456588e8733e2406f8ed (127.27.17.129:37957): tablet cabe053775754da49c6bc46e79d13bf3 (table test-workload [id=7dc4ebdd05fa450f98dda71f1548b724]) successfully deleted
I20260810 22:04:29.309384 29015 catalog_manager.cc:5495] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2 reported cstate change: config changed from index 2 to 3, NON_VOTER 6b67f5fd3c9943a7bf77411798547a55 (127.27.17.130) added. New cstate: current_term: 2 leader_uuid: "21ef9e69018e484a8dbf082e54bbc07a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: NON_VOTER last_known_addr { host: "127.27.17.130" port: 44819 } attrs { promote: true } } }
I20260810 22:04:29.311319 29000 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cabe053775754da49c6bc46e79d13bf3 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20260810 22:04:29.324225 29712 consensus_peers.cc:487] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a -> Peer 6b67f5fd3c9943a7bf77411798547a55 (127.27.17.130:44819): Couldn't send request to peer 6b67f5fd3c9943a7bf77411798547a55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cabe053775754da49c6bc46e79d13bf3. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:29.400066 29771 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 22:04:29.405486 29374 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 22:04:29.420368 30117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 22:04:29.422152 29577 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 22:04:29.591955 30271 ts_tablet_manager.cc:927] T dcfa54f1b7ef4b0b95226226bd78c40b P 21ef9e69018e484a8dbf082e54bbc07a: Initiating tablet copy from peer 6b67f5fd3c9943a7bf77411798547a55 (127.27.17.130:44819)
I20260810 22:04:29.634298 30271 tablet_copy_client.cc:316] T dcfa54f1b7ef4b0b95226226bd78c40b P 21ef9e69018e484a8dbf082e54bbc07a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.17.130:44819
I20260810 22:04:29.636528 29432 tablet_copy_service.cc:140] P 6b67f5fd3c9943a7bf77411798547a55: Received BeginTabletCopySession request for tablet dcfa54f1b7ef4b0b95226226bd78c40b from peer 21ef9e69018e484a8dbf082e54bbc07a ({username='slave'} at 127.27.17.132:56011)
I20260810 22:04:29.637096 29432 tablet_copy_service.cc:161] P 6b67f5fd3c9943a7bf77411798547a55: Beginning new tablet copy session on tablet dcfa54f1b7ef4b0b95226226bd78c40b from peer 21ef9e69018e484a8dbf082e54bbc07a at {username='slave'} at 127.27.17.132:56011: session id = 21ef9e69018e484a8dbf082e54bbc07a-dcfa54f1b7ef4b0b95226226bd78c40b
I20260810 22:04:29.645231 29432 tablet_copy_source_session.cc:205] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:04:29.669274 30271 tablet_copy_client.cc:791] T dcfa54f1b7ef4b0b95226226bd78c40b P 21ef9e69018e484a8dbf082e54bbc07a: tablet copy: Starting download of 0 data blocks...
I20260810 22:04:29.670007 30271 tablet_copy_client.cc:655] T dcfa54f1b7ef4b0b95226226bd78c40b P 21ef9e69018e484a8dbf082e54bbc07a: tablet copy: Starting download of 1 WAL segments...
I20260810 22:04:29.674747 30271 tablet_copy_client.cc:531] T dcfa54f1b7ef4b0b95226226bd78c40b P 21ef9e69018e484a8dbf082e54bbc07a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:04:29.683470 30271 tablet_bootstrap.cc:492] T dcfa54f1b7ef4b0b95226226bd78c40b P 21ef9e69018e484a8dbf082e54bbc07a: Bootstrap starting.
I20260810 22:04:29.709172 30278 ts_tablet_manager.cc:927] T cabe053775754da49c6bc46e79d13bf3 P 6b67f5fd3c9943a7bf77411798547a55: Initiating tablet copy from peer 21ef9e69018e484a8dbf082e54bbc07a (127.27.17.132:38435)
I20260810 22:04:29.717046 30278 tablet_copy_client.cc:316] T cabe053775754da49c6bc46e79d13bf3 P 6b67f5fd3c9943a7bf77411798547a55: tablet copy: Beginning tablet copy session from remote peer at address 127.27.17.132:38435
I20260810 22:04:29.719180 29826 tablet_copy_service.cc:140] P 21ef9e69018e484a8dbf082e54bbc07a: Received BeginTabletCopySession request for tablet cabe053775754da49c6bc46e79d13bf3 from peer 6b67f5fd3c9943a7bf77411798547a55 ({username='slave'} at 127.27.17.130:49581)
I20260810 22:04:29.719784 29826 tablet_copy_service.cc:161] P 21ef9e69018e484a8dbf082e54bbc07a: Beginning new tablet copy session on tablet cabe053775754da49c6bc46e79d13bf3 from peer 6b67f5fd3c9943a7bf77411798547a55 at {username='slave'} at 127.27.17.130:49581: session id = 6b67f5fd3c9943a7bf77411798547a55-cabe053775754da49c6bc46e79d13bf3
I20260810 22:04:29.725204 30271 tablet_bootstrap.cc:492] T dcfa54f1b7ef4b0b95226226bd78c40b P 21ef9e69018e484a8dbf082e54bbc07a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:04:29.726141 30271 tablet_bootstrap.cc:492] T dcfa54f1b7ef4b0b95226226bd78c40b P 21ef9e69018e484a8dbf082e54bbc07a: Bootstrap complete.
I20260810 22:04:29.726771 30271 ts_tablet_manager.cc:1397] T dcfa54f1b7ef4b0b95226226bd78c40b P 21ef9e69018e484a8dbf082e54bbc07a: Time spent bootstrapping tablet: real 0.044s user 0.017s sys 0.009s
I20260810 22:04:29.728068 29826 tablet_copy_source_session.cc:205] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a: Tablet Copy: opened 0 blocks and 1 log segments
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d7577a8b027a4a80ad0f7b2f460658b7 | 127.27.17.190:35505 | HEALTHY
Unusual flags for Master:
I20260810 22:04:29.729452 30271 raft_consensus.cc:348] T dcfa54f1b7ef4b0b95226226bd78c40b P 21ef9e69018e484a8dbf082e54bbc07a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: NON_VOTER last_known_addr { host: "127.27.17.132" port: 38435 } attrs { promote: true } }
I20260810 22:04:29.730275 30271 raft_consensus.cc:729] T dcfa54f1b7ef4b0b95226226bd78c40b P 21ef9e69018e484a8dbf082e54bbc07a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21ef9e69018e484a8dbf082e54bbc07a, State: Initialized, Role: LEARNER
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
I20260810 22:04:29.730993 30271 consensus_queue.cc:260] T dcfa54f1b7ef4b0b95226226bd78c40b P 21ef9e69018e484a8dbf082e54bbc07a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: NON_VOTER last_known_addr { host: "127.27.17.132" port: 38435 } attrs { promote: true } }
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:41977 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
21ef9e69018e484a8dbf082e54bbc07a | 127.27.17.132:38435 | HEALTHY | <none> | 1 | 0
6b67f5fd3c9943a7bf77411798547a55 | 127.27.17.130:44819 | HEALTHY | <none> | 3 | 0
9eb15a3b74af456588e8733e2406f8ed | 127.27.17.129:37957 | HEALTHY | <none> | 0 | 0
cfe74abd0282488f9a8ce6c152b761b2 | 127.27.17.131:38077 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
I20260810 22:04:29.735960 30271 ts_tablet_manager.cc:1428] T dcfa54f1b7ef4b0b95226226bd78c40b P 21ef9e69018e484a8dbf082e54bbc07a: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260810 22:04:29.738008 29432 tablet_copy_service.cc:342] P 6b67f5fd3c9943a7bf77411798547a55: Request end of tablet copy session 21ef9e69018e484a8dbf082e54bbc07a-dcfa54f1b7ef4b0b95226226bd78c40b received from {username='slave'} at 127.27.17.132:56011
I20260810 22:04:29.738415 29432 tablet_copy_service.cc:433] P 6b67f5fd3c9943a7bf77411798547a55: ending tablet copy session 21ef9e69018e484a8dbf082e54bbc07a-dcfa54f1b7ef4b0b95226226bd78c40b on tablet dcfa54f1b7ef4b0b95226226bd78c40b with peer 21ef9e69018e484a8dbf082e54bbc07a
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.27.17.129:37957
env_inject_eio | 1 | hidden | 127.27.17.129:37957
env_inject_eio_globs | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-1/** | hidden | 127.27.17.129:37957
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.27.17.129 | experimental | 127.27.17.129:37957
local_ip_for_outbound_sockets | 127.27.17.130 | experimental | 127.27.17.130:44819
local_ip_for_outbound_sockets | 127.27.17.131 | experimental | 127.27.17.131:38077
local_ip_for_outbound_sockets | 127.27.17.132 | experimental | 127.27.17.132:38435
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.27.17.129:37957
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.27.17.130:44819
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.27.17.131:38077
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786399438989427-27718-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.27.17.132:38435
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:41977 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
I20260810 22:04:29.747699 30278 tablet_copy_client.cc:791] T cabe053775754da49c6bc46e79d13bf3 P 6b67f5fd3c9943a7bf77411798547a55: tablet copy: Starting download of 0 data blocks...
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
I20260810 22:04:29.748292 30278 tablet_copy_client.cc:655] T cabe053775754da49c6bc46e79d13bf3 P 6b67f5fd3c9943a7bf77411798547a55: tablet copy: Starting download of 1 WAL segments...
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 5 | 5 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 4
Median | 4
Third Quartile | 5
Maximum | 5
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 9eb15a3b74af456588e8733e2406f8ed | 127.27.17.129:37957 | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260810 22:04:29.752418 30278 tablet_copy_client.cc:531] T cabe053775754da49c6bc46e79d13bf3 P 6b67f5fd3c9943a7bf77411798547a55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:04:29.760726 30278 tablet_bootstrap.cc:492] T cabe053775754da49c6bc46e79d13bf3 P 6b67f5fd3c9943a7bf77411798547a55: Bootstrap starting.
I20260810 22:04:29.769575 27718 log_verifier.cc:126] Checking tablet 7ba3cc9bd6c9412dba778d21fd65181c
I20260810 22:04:29.799500 30278 tablet_bootstrap.cc:492] T cabe053775754da49c6bc46e79d13bf3 P 6b67f5fd3c9943a7bf77411798547a55: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:04:29.801312 30278 tablet_bootstrap.cc:492] T cabe053775754da49c6bc46e79d13bf3 P 6b67f5fd3c9943a7bf77411798547a55: Bootstrap complete.
I20260810 22:04:29.802102 30278 ts_tablet_manager.cc:1397] T cabe053775754da49c6bc46e79d13bf3 P 6b67f5fd3c9943a7bf77411798547a55: Time spent bootstrapping tablet: real 0.042s user 0.022s sys 0.018s
I20260810 22:04:29.804639 30278 raft_consensus.cc:348] T cabe053775754da49c6bc46e79d13bf3 P 6b67f5fd3c9943a7bf77411798547a55 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: NON_VOTER last_known_addr { host: "127.27.17.130" port: 44819 } attrs { promote: true } }
I20260810 22:04:29.805219 30278 raft_consensus.cc:729] T cabe053775754da49c6bc46e79d13bf3 P 6b67f5fd3c9943a7bf77411798547a55 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 6b67f5fd3c9943a7bf77411798547a55, State: Initialized, Role: LEARNER
I20260810 22:04:29.805765 30278 consensus_queue.cc:260] T cabe053775754da49c6bc46e79d13bf3 P 6b67f5fd3c9943a7bf77411798547a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: NON_VOTER last_known_addr { host: "127.27.17.130" port: 44819 } attrs { promote: true } }
I20260810 22:04:29.807875 30278 ts_tablet_manager.cc:1428] T cabe053775754da49c6bc46e79d13bf3 P 6b67f5fd3c9943a7bf77411798547a55: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260810 22:04:29.810006 29826 tablet_copy_service.cc:342] P 21ef9e69018e484a8dbf082e54bbc07a: Request end of tablet copy session 6b67f5fd3c9943a7bf77411798547a55-cabe053775754da49c6bc46e79d13bf3 received from {username='slave'} at 127.27.17.130:49581
I20260810 22:04:29.810423 29826 tablet_copy_service.cc:433] P 21ef9e69018e484a8dbf082e54bbc07a: ending tablet copy session 6b67f5fd3c9943a7bf77411798547a55-cabe053775754da49c6bc46e79d13bf3 on tablet cabe053775754da49c6bc46e79d13bf3 with peer 6b67f5fd3c9943a7bf77411798547a55
I20260810 22:04:29.863600 27718 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 7ba3cc9bd6c9412dba778d21fd65181c
I20260810 22:04:29.864012 27718 log_verifier.cc:126] Checking tablet 8690baf5d2f54129846bba669f5c6f3d
E20260810 22:04:29.935392 30197 env_posix.cc:432] I/O error, context: INJECTED FAILURE
I20260810 22:04:29.952004 27718 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 8690baf5d2f54129846bba669f5c6f3d
I20260810 22:04:29.952282 27718 log_verifier.cc:126] Checking tablet cabe053775754da49c6bc46e79d13bf3
I20260810 22:04:30.022873 27718 log_verifier.cc:177] Verified matching terms for 3 ops in tablet cabe053775754da49c6bc46e79d13bf3
I20260810 22:04:30.023092 27718 log_verifier.cc:126] Checking tablet dcfa54f1b7ef4b0b95226226bd78c40b
I20260810 22:04:30.092327 27718 log_verifier.cc:177] Verified matching terms for 3 ops in tablet dcfa54f1b7ef4b0b95226226bd78c40b
I20260810 22:04:30.092568 27718 log_verifier.cc:126] Checking tablet fff845c32f1f4b81b6b8d0052cf06968
I20260810 22:04:30.097476 29800 raft_consensus.cc:1203] T dcfa54f1b7ef4b0b95226226bd78c40b P 21ef9e69018e484a8dbf082e54bbc07a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20260810 22:04:30.160647 27718 log_verifier.cc:177] Verified matching terms for 1 ops in tablet fff845c32f1f4b81b6b8d0052cf06968
I20260810 22:04:30.207505 29402 raft_consensus.cc:1203] T cabe053775754da49c6bc46e79d13bf3 P 6b67f5fd3c9943a7bf77411798547a55 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260810 22:04:30.589702 30306 raft_consensus.cc:1050] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a: attempting to promote NON_VOTER 6b67f5fd3c9943a7bf77411798547a55 to VOTER
I20260810 22:04:30.591848 30306 consensus_queue.cc:237] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } attrs { promote: false } }
I20260810 22:04:30.612250 29402 raft_consensus.cc:1261] T cabe053775754da49c6bc46e79d13bf3 P 6b67f5fd3c9943a7bf77411798547a55 [term 2 LEARNER]: Refusing update from remote peer 21ef9e69018e484a8dbf082e54bbc07a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260810 22:04:30.617651 30003 consensus_queue.cc:1035] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 22:04:30.627014 29607 raft_consensus.cc:1261] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2 [term 2 FOLLOWER]: Refusing update from remote peer 21ef9e69018e484a8dbf082e54bbc07a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260810 22:04:30.628509 30302 consensus_queue.cc:1035] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 22:04:30.644268 30003 raft_consensus.cc:2932] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 6b67f5fd3c9943a7bf77411798547a55 (127.27.17.130) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } attrs { promote: false } } }
I20260810 22:04:30.678370 29607 raft_consensus.cc:2932] T cabe053775754da49c6bc46e79d13bf3 P cfe74abd0282488f9a8ce6c152b761b2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 6b67f5fd3c9943a7bf77411798547a55 (127.27.17.130) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } attrs { promote: false } } }
I20260810 22:04:30.685667 29015 catalog_manager.cc:5495] T cabe053775754da49c6bc46e79d13bf3 P 21ef9e69018e484a8dbf082e54bbc07a reported cstate change: config changed from index 3 to 4, 6b67f5fd3c9943a7bf77411798547a55 (127.27.17.130) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "21ef9e69018e484a8dbf082e54bbc07a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260810 22:04:30.722191 29402 raft_consensus.cc:2932] T cabe053775754da49c6bc46e79d13bf3 P 6b67f5fd3c9943a7bf77411798547a55 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 6b67f5fd3c9943a7bf77411798547a55 (127.27.17.130) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } } peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } attrs { promote: false } } }
I20260810 22:04:30.750119 30320 raft_consensus.cc:1050] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55: attempting to promote NON_VOTER 21ef9e69018e484a8dbf082e54bbc07a to VOTER
I20260810 22:04:30.759325 30320 consensus_queue.cc:237] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } attrs { promote: false } }
I20260810 22:04:30.779213 29800 raft_consensus.cc:1261] T dcfa54f1b7ef4b0b95226226bd78c40b P 21ef9e69018e484a8dbf082e54bbc07a [term 1 LEARNER]: Refusing update from remote peer 6b67f5fd3c9943a7bf77411798547a55: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 22:04:30.780917 29932 consensus_queue.cc:1035] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 22:04:30.781795 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 29965
I20260810 22:04:30.799610 29608 raft_consensus.cc:1261] T dcfa54f1b7ef4b0b95226226bd78c40b P cfe74abd0282488f9a8ce6c152b761b2 [term 1 FOLLOWER]: Refusing update from remote peer 6b67f5fd3c9943a7bf77411798547a55: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 22:04:30.801324 29932 consensus_queue.cc:1035] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 22:04:30.807791 30319 raft_consensus.cc:2932] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 21ef9e69018e484a8dbf082e54bbc07a (127.27.17.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } attrs { promote: false } } }
I20260810 22:04:30.815625 29799 raft_consensus.cc:2932] T dcfa54f1b7ef4b0b95226226bd78c40b P 21ef9e69018e484a8dbf082e54bbc07a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 21ef9e69018e484a8dbf082e54bbc07a (127.27.17.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } attrs { promote: false } } }
I20260810 22:04:30.823475 29013 catalog_manager.cc:5495] T dcfa54f1b7ef4b0b95226226bd78c40b P 6b67f5fd3c9943a7bf77411798547a55 reported cstate change: config changed from index 3 to 4, 21ef9e69018e484a8dbf082e54bbc07a (127.27.17.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6b67f5fd3c9943a7bf77411798547a55" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 22:04:30.828466 29608 raft_consensus.cc:2932] T dcfa54f1b7ef4b0b95226226bd78c40b P cfe74abd0282488f9a8ce6c152b761b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 21ef9e69018e484a8dbf082e54bbc07a (127.27.17.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b67f5fd3c9943a7bf77411798547a55" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44819 } } peers { permanent_uuid: "cfe74abd0282488f9a8ce6c152b761b2" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38077 } } peers { permanent_uuid: "21ef9e69018e484a8dbf082e54bbc07a" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 38435 } attrs { promote: false } } }
I20260810 22:04:30.944399 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 29286
I20260810 22:04:31.042584 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 29481
I20260810 22:04:31.091008 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 29681
I20260810 22:04:31.133442 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 28964
2026-08-10T22:04:31Z chronyd exiting
[ OK ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/1 (15178 ms)
[ RUN ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/2
2026-08-10T22:04:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:04:31Z Disabled control of system clock
I20260810 22:04:31.255541 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.190:33411
--webserver_interface=127.27.17.190
--webserver_port=0
--builtin_ntp_servers=127.27.17.148:42125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.190:33411
--raft_prepare_replacement_before_eviction=false with env {}
W20260810 22:04:31.571468 30335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:31.572160 30335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:31.572674 30335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:31.604439 30335 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260810 22:04:31.604898 30335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:04:31.605195 30335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:31.605472 30335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:04:31.605737 30335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:04:31.641738 30335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:42125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.190:33411
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.190:33411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
--webserver_interface=127.27.17.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:31.643024 30335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:04:31.644655 30335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:31.662133 30348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:31.662807 30344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:31.664002 30335 server_base.cc:1034] running on GCE node
W20260810 22:04:31.662120 30345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:32.845292 30335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:32.848065 30335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:32.849453 30335 hybrid_clock.cc:648] HybridClock initialized: now 1786399472849426 us; error 78 us; skew 500 ppm
I20260810 22:04:32.850325 30335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:32.863108 30335 webserver.cc:460] Webserver started at http://127.27.17.190:34241/ using document root <none> and password file <none>
I20260810 22:04:32.864774 30335 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:32.865058 30335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:04:32.865840 30335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:04:32.871691 30335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-0/instance:
uuid: "22693aa679da4011b230e0e3fcf343fe"
format_stamp: "Formatted at 2026-08-10 22:04:32 on dist-test-slave-bw88"
I20260810 22:04:32.873014 30335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-1/instance:
uuid: "22693aa679da4011b230e0e3fcf343fe"
format_stamp: "Formatted at 2026-08-10 22:04:32 on dist-test-slave-bw88"
I20260810 22:04:32.874078 30335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-2/instance:
uuid: "22693aa679da4011b230e0e3fcf343fe"
format_stamp: "Formatted at 2026-08-10 22:04:32 on dist-test-slave-bw88"
I20260810 22:04:32.875037 30335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-3/instance:
uuid: "22693aa679da4011b230e0e3fcf343fe"
format_stamp: "Formatted at 2026-08-10 22:04:32 on dist-test-slave-bw88"
I20260810 22:04:32.876070 30335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-4/instance:
uuid: "22693aa679da4011b230e0e3fcf343fe"
format_stamp: "Formatted at 2026-08-10 22:04:32 on dist-test-slave-bw88"
I20260810 22:04:32.877069 30335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/wal/instance:
uuid: "22693aa679da4011b230e0e3fcf343fe"
format_stamp: "Formatted at 2026-08-10 22:04:32 on dist-test-slave-bw88"
I20260810 22:04:32.902109 30335 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.015s sys 0.008s
I20260810 22:04:32.914110 30368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:32.915041 30368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:32.915108 30369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:32.916596 30369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:32.916980 30370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:32.919322 30335 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.005s
I20260810 22:04:32.919806 30335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/wal
uuid: "22693aa679da4011b230e0e3fcf343fe"
format_stamp: "Formatted at 2026-08-10 22:04:32 on dist-test-slave-bw88"
I20260810 22:04:32.920372 30335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:04:32.959800 30335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:32.961294 30335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:32.961747 30335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:33.031476 30335 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.190:33411
I20260810 22:04:33.031945 30438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.190:33411 every 8 connection(s)
I20260810 22:04:33.034545 30335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
I20260810 22:04:33.043737 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 30335
I20260810 22:04:33.044279 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/wal/instance
I20260810 22:04:33.061468 30440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe: Bootstrap starting.
I20260810 22:04:33.067283 30440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:33.069747 30440 log.cc:824] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:33.074681 30440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe: No bootstrap required, opened a new log
I20260810 22:04:33.099267 30440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22693aa679da4011b230e0e3fcf343fe" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 33411 } }
I20260810 22:04:33.099946 30440 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:33.100157 30440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22693aa679da4011b230e0e3fcf343fe, State: Initialized, Role: FOLLOWER
I20260810 22:04:33.100880 30440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22693aa679da4011b230e0e3fcf343fe" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 33411 } }
I20260810 22:04:33.101459 30440 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:04:33.101697 30440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:04:33.101956 30440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:33.106456 30440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22693aa679da4011b230e0e3fcf343fe" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 33411 } }
I20260810 22:04:33.107290 30440 leader_election.cc:302] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22693aa679da4011b230e0e3fcf343fe; no voters:
I20260810 22:04:33.109081 30440 leader_election.cc:288] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:04:33.109398 30448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:33.112210 30448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe [term 1 LEADER]: Becoming Leader. State: Replica: 22693aa679da4011b230e0e3fcf343fe, State: Running, Role: LEADER
I20260810 22:04:33.112999 30448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22693aa679da4011b230e0e3fcf343fe" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 33411 } }
I20260810 22:04:33.113905 30440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:04:33.123508 30450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22693aa679da4011b230e0e3fcf343fe. Latest consensus state: current_term: 1 leader_uuid: "22693aa679da4011b230e0e3fcf343fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22693aa679da4011b230e0e3fcf343fe" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 33411 } } }
I20260810 22:04:33.124251 30450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe [sys.catalog]: This master's current role is: LEADER
I20260810 22:04:33.124172 30449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22693aa679da4011b230e0e3fcf343fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22693aa679da4011b230e0e3fcf343fe" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 33411 } } }
I20260810 22:04:33.124825 30449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe [sys.catalog]: This master's current role is: LEADER
I20260810 22:04:33.127869 30457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:04:33.141636 30457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:04:33.163085 30457 catalog_manager.cc:1347] Generated new cluster ID: d3c11de46085427c92b9d89ab46d4301
I20260810 22:04:33.163364 30457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:04:33.183549 30457 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:04:33.185444 30457 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:04:33.198351 30457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22693aa679da4011b230e0e3fcf343fe: Generated new TSK 0
I20260810 22:04:33.199273 30457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:04:33.217154 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.129:0
--local_ip_for_outbound_sockets=127.27.17.129
--webserver_interface=127.27.17.129
--webserver_port=0
--tserver_master_addrs=127.27.17.190:33411
--builtin_ntp_servers=127.27.17.148:42125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20260810 22:04:33.548527 30475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:33.549109 30475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:33.549696 30475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:33.582621 30475 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260810 22:04:33.583084 30475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:33.583958 30475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.129
I20260810 22:04:33.618700 30475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:42125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.27.17.129
--webserver_port=0
--tserver_master_addrs=127.27.17.190:33411
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.129
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:33.620029 30475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:04:33.621794 30475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:33.642627 30486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:33.644436 30483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:33.642774 30488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:34.862174 30487 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 22:04:34.862293 30475 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 22:04:34.866300 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:34.869246 30475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:34.870770 30475 hybrid_clock.cc:648] HybridClock initialized: now 1786399474870737 us; error 49 us; skew 500 ppm
I20260810 22:04:34.871821 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:34.879921 30475 webserver.cc:460] Webserver started at http://127.27.17.129:37987/ using document root <none> and password file <none>
I20260810 22:04:34.881946 30475 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:34.882215 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:04:34.883052 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:04:34.890101 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-0/instance:
uuid: "842b26ed57e14f5784632195da785fc8"
format_stamp: "Formatted at 2026-08-10 22:04:34 on dist-test-slave-bw88"
I20260810 22:04:34.891744 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-1/instance:
uuid: "842b26ed57e14f5784632195da785fc8"
format_stamp: "Formatted at 2026-08-10 22:04:34 on dist-test-slave-bw88"
I20260810 22:04:34.892905 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-2/instance:
uuid: "842b26ed57e14f5784632195da785fc8"
format_stamp: "Formatted at 2026-08-10 22:04:34 on dist-test-slave-bw88"
I20260810 22:04:34.894109 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-3/instance:
uuid: "842b26ed57e14f5784632195da785fc8"
format_stamp: "Formatted at 2026-08-10 22:04:34 on dist-test-slave-bw88"
I20260810 22:04:34.895363 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-4/instance:
uuid: "842b26ed57e14f5784632195da785fc8"
format_stamp: "Formatted at 2026-08-10 22:04:34 on dist-test-slave-bw88"
I20260810 22:04:34.896457 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/wal/instance:
uuid: "842b26ed57e14f5784632195da785fc8"
format_stamp: "Formatted at 2026-08-10 22:04:34 on dist-test-slave-bw88"
I20260810 22:04:34.920830 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.012s sys 0.014s
I20260810 22:04:34.929967 30501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:34.930771 30501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:34.931504 30501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:34.931932 30501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:34.931921 30502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:34.935009 30475 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.005s
I20260810 22:04:34.935364 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/wal
uuid: "842b26ed57e14f5784632195da785fc8"
format_stamp: "Formatted at 2026-08-10 22:04:34 on dist-test-slave-bw88"
I20260810 22:04:34.935774 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:04:34.989178 30475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:34.990795 30475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:34.991279 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:34.994098 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:04:34.998782 30475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:04:34.999099 30475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:34.999423 30475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:04:34.999660 30475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:35.147648 30475 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.129:37815
I20260810 22:04:35.148135 30681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.129:37815 every 8 connection(s)
I20260810 22:04:35.150784 30475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
I20260810 22:04:35.158217 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 30475
I20260810 22:04:35.158756 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/wal/instance
I20260810 22:04:35.166724 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.130:0
--local_ip_for_outbound_sockets=127.27.17.130
--webserver_interface=127.27.17.130
--webserver_port=0
--tserver_master_addrs=127.27.17.190:33411
--builtin_ntp_servers=127.27.17.148:42125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20260810 22:04:35.179641 30682 heartbeater.cc:347] Connected to a master server at 127.27.17.190:33411
I20260810 22:04:35.180197 30682 heartbeater.cc:464] Registering TS with master...
I20260810 22:04:35.181712 30682 heartbeater.cc:511] Master 127.27.17.190:33411 requested a full tablet report, sending...
I20260810 22:04:35.184767 30393 ts_manager.cc:194] Registered new tserver with Master: 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815)
I20260810 22:04:35.187096 30393 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.129:37691
W20260810 22:04:35.493621 30687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:35.494168 30687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:35.494699 30687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:35.525811 30687 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260810 22:04:35.526242 30687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:35.527058 30687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.130
I20260810 22:04:35.562562 30687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:42125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
--webserver_interface=127.27.17.130
--webserver_port=0
--tserver_master_addrs=127.27.17.190:33411
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.130
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:35.563862 30687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:04:35.565589 30687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:35.585956 30697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:36.192251 30682 heartbeater.cc:503] Master 127.27.17.190:33411 was elected leader, sending a full tablet report...
W20260810 22:04:35.586642 30702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:35.586665 30699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:35.588315 30687 server_base.cc:1034] running on GCE node
I20260810 22:04:36.772822 30687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:36.775851 30687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:36.777447 30687 hybrid_clock.cc:648] HybridClock initialized: now 1786399476777402 us; error 50 us; skew 500 ppm
I20260810 22:04:36.778317 30687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:36.785239 30687 webserver.cc:460] Webserver started at http://127.27.17.130:42249/ using document root <none> and password file <none>
I20260810 22:04:36.786798 30687 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:36.787007 30687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:04:36.787647 30687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:04:36.792537 30687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-0/instance:
uuid: "219a405626974999a6bdaa700b03120e"
format_stamp: "Formatted at 2026-08-10 22:04:36 on dist-test-slave-bw88"
I20260810 22:04:36.793627 30687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-1/instance:
uuid: "219a405626974999a6bdaa700b03120e"
format_stamp: "Formatted at 2026-08-10 22:04:36 on dist-test-slave-bw88"
I20260810 22:04:36.794466 30687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-2/instance:
uuid: "219a405626974999a6bdaa700b03120e"
format_stamp: "Formatted at 2026-08-10 22:04:36 on dist-test-slave-bw88"
I20260810 22:04:36.795275 30687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-3/instance:
uuid: "219a405626974999a6bdaa700b03120e"
format_stamp: "Formatted at 2026-08-10 22:04:36 on dist-test-slave-bw88"
I20260810 22:04:36.796134 30687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-4/instance:
uuid: "219a405626974999a6bdaa700b03120e"
format_stamp: "Formatted at 2026-08-10 22:04:36 on dist-test-slave-bw88"
I20260810 22:04:36.796944 30687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/wal/instance:
uuid: "219a405626974999a6bdaa700b03120e"
format_stamp: "Formatted at 2026-08-10 22:04:36 on dist-test-slave-bw88"
I20260810 22:04:36.820360 30687 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.017s sys 0.005s
I20260810 22:04:36.828819 30718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:36.829675 30718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:36.830193 30718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:36.830525 30719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:36.830698 30718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:36.833528 30687 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.004s
I20260810 22:04:36.833875 30687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/wal
uuid: "219a405626974999a6bdaa700b03120e"
format_stamp: "Formatted at 2026-08-10 22:04:36 on dist-test-slave-bw88"
I20260810 22:04:36.834285 30687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:04:36.892858 30687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:36.894464 30687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:36.894939 30687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:36.897632 30687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:04:36.902300 30687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:04:36.902549 30687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:36.902819 30687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:04:36.903012 30687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:37.054291 30687 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.130:36727
I20260810 22:04:37.054826 30871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.130:36727 every 8 connection(s)
I20260810 22:04:37.057458 30687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
I20260810 22:04:37.066358 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 30687
I20260810 22:04:37.066938 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/wal/instance
I20260810 22:04:37.074764 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.131:0
--local_ip_for_outbound_sockets=127.27.17.131
--webserver_interface=127.27.17.131
--webserver_port=0
--tserver_master_addrs=127.27.17.190:33411
--builtin_ntp_servers=127.27.17.148:42125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20260810 22:04:37.086721 30872 heartbeater.cc:347] Connected to a master server at 127.27.17.190:33411
I20260810 22:04:37.087293 30872 heartbeater.cc:464] Registering TS with master...
I20260810 22:04:37.088634 30872 heartbeater.cc:511] Master 127.27.17.190:33411 requested a full tablet report, sending...
I20260810 22:04:37.091421 30393 ts_manager.cc:194] Registered new tserver with Master: 219a405626974999a6bdaa700b03120e (127.27.17.130:36727)
I20260810 22:04:37.092799 30393 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.130:41169
W20260810 22:04:37.405004 30877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:37.405592 30877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:37.406106 30877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:37.437408 30877 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260810 22:04:37.437852 30877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:37.438666 30877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.131
I20260810 22:04:37.473754 30877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:42125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
--webserver_interface=127.27.17.131
--webserver_port=0
--tserver_master_addrs=127.27.17.190:33411
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.131
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:37.475031 30877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:04:37.476643 30877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:37.497093 30891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:38.096486 30872 heartbeater.cc:503] Master 127.27.17.190:33411 was elected leader, sending a full tablet report...
W20260810 22:04:37.497139 30890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:37.497292 30894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:38.717808 30892 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1218 milliseconds
I20260810 22:04:38.717927 30877 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 22:04:38.719257 30877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:38.722123 30877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:38.723552 30877 hybrid_clock.cc:648] HybridClock initialized: now 1786399478723530 us; error 57 us; skew 500 ppm
I20260810 22:04:38.724336 30877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:38.736414 30877 webserver.cc:460] Webserver started at http://127.27.17.131:44547/ using document root <none> and password file <none>
I20260810 22:04:38.737985 30877 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:38.738217 30877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:04:38.738879 30877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:04:38.743999 30877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-0/instance:
uuid: "bd15e90d358641be9ce4e5563f57de30"
format_stamp: "Formatted at 2026-08-10 22:04:38 on dist-test-slave-bw88"
I20260810 22:04:38.745153 30877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-1/instance:
uuid: "bd15e90d358641be9ce4e5563f57de30"
format_stamp: "Formatted at 2026-08-10 22:04:38 on dist-test-slave-bw88"
I20260810 22:04:38.746105 30877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-2/instance:
uuid: "bd15e90d358641be9ce4e5563f57de30"
format_stamp: "Formatted at 2026-08-10 22:04:38 on dist-test-slave-bw88"
I20260810 22:04:38.746980 30877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-3/instance:
uuid: "bd15e90d358641be9ce4e5563f57de30"
format_stamp: "Formatted at 2026-08-10 22:04:38 on dist-test-slave-bw88"
I20260810 22:04:38.747854 30877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-4/instance:
uuid: "bd15e90d358641be9ce4e5563f57de30"
format_stamp: "Formatted at 2026-08-10 22:04:38 on dist-test-slave-bw88"
I20260810 22:04:38.748729 30877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/wal/instance:
uuid: "bd15e90d358641be9ce4e5563f57de30"
format_stamp: "Formatted at 2026-08-10 22:04:38 on dist-test-slave-bw88"
I20260810 22:04:38.772135 30877 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.025s sys 0.000s
I20260810 22:04:38.780105 30906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:38.780735 30906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:38.781672 30906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:38.781888 30907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:38.782018 30906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:38.784005 30877 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.006s
I20260810 22:04:38.784308 30877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/wal
uuid: "bd15e90d358641be9ce4e5563f57de30"
format_stamp: "Formatted at 2026-08-10 22:04:38 on dist-test-slave-bw88"
I20260810 22:04:38.784668 30877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:04:38.833244 30877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:38.834875 30877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:38.835304 30877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:38.837987 30877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:04:38.842764 30877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:04:38.842998 30877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:38.843271 30877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:04:38.843420 30877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:38.987857 30877 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.131:35619
I20260810 22:04:38.988327 31053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.131:35619 every 8 connection(s)
I20260810 22:04:38.991111 30877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
I20260810 22:04:39.001549 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 30877
I20260810 22:04:39.002040 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/wal/instance
I20260810 22:04:39.015249 31054 heartbeater.cc:347] Connected to a master server at 127.27.17.190:33411
I20260810 22:04:39.015769 31054 heartbeater.cc:464] Registering TS with master...
I20260810 22:04:39.016680 31054 heartbeater.cc:511] Master 127.27.17.190:33411 requested a full tablet report, sending...
I20260810 22:04:39.018959 30393 ts_manager.cc:194] Registered new tserver with Master: bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619)
I20260810 22:04:39.020913 30393 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.131:53939
I20260810 22:04:39.022858 27718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:04:39.054488 27718 test_util.cc:268] Using random seed: -1973403522
I20260810 22:04:39.103219 30392 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40614:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\231\231\231\031\004\001\0002333\004\001\000\313\314\314L\004\001\000dfff"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 22:04:39.182314 30962 tablet_service.cc:1467] Processing CreateTablet for tablet acf4ac19b8ea418aa7827d0c3df14ae0 (DEFAULT_TABLE table=test-workload [id=7bdc63e81a8f42aca69adc75753bc653]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260810 22:04:39.183799 30968 tablet_service.cc:1467] Processing CreateTablet for tablet 793f84134c904805bcf80ae4d941d6d4 (DEFAULT_TABLE table=test-workload [id=7bdc63e81a8f42aca69adc75753bc653]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260810 22:04:39.186204 30967 tablet_service.cc:1467] Processing CreateTablet for tablet b71556c430684bc889e3c0f013e7f38b (DEFAULT_TABLE table=test-workload [id=7bdc63e81a8f42aca69adc75753bc653]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260810 22:04:39.189256 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 793f84134c904805bcf80ae4d941d6d4 (DEFAULT_TABLE table=test-workload [id=7bdc63e81a8f42aca69adc75753bc653]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260810 22:04:39.189584 30965 tablet_service.cc:1467] Processing CreateTablet for tablet f969babe0c5c4ae0a8a8e7162d361173 (DEFAULT_TABLE table=test-workload [id=7bdc63e81a8f42aca69adc75753bc653]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260810 22:04:39.191196 30779 tablet_service.cc:1467] Processing CreateTablet for tablet acf4ac19b8ea418aa7827d0c3df14ae0 (DEFAULT_TABLE table=test-workload [id=7bdc63e81a8f42aca69adc75753bc653]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260810 22:04:39.194725 30782 tablet_service.cc:1467] Processing CreateTablet for tablet b71556c430684bc889e3c0f013e7f38b (DEFAULT_TABLE table=test-workload [id=7bdc63e81a8f42aca69adc75753bc653]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260810 22:04:39.195640 30969 tablet_service.cc:1467] Processing CreateTablet for tablet 21df603d01494efd960235764bf9e19b (DEFAULT_TABLE table=test-workload [id=7bdc63e81a8f42aca69adc75753bc653]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260810 22:04:39.188459 30781 tablet_service.cc:1467] Processing CreateTablet for tablet f969babe0c5c4ae0a8a8e7162d361173 (DEFAULT_TABLE table=test-workload [id=7bdc63e81a8f42aca69adc75753bc653]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260810 22:04:39.187379 30785 tablet_service.cc:1467] Processing CreateTablet for tablet 21df603d01494efd960235764bf9e19b (DEFAULT_TABLE table=test-workload [id=7bdc63e81a8f42aca69adc75753bc653]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260810 22:04:39.223223 31091 tablet_bootstrap.cc:492] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30: Bootstrap starting.
I20260810 22:04:39.231012 31091 tablet_bootstrap.cc:654] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:39.243875 31091 log.cc:824] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:39.253587 31092 tablet_bootstrap.cc:492] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e: Bootstrap starting.
I20260810 22:04:39.254276 30571 tablet_service.cc:1467] Processing CreateTablet for tablet b71556c430684bc889e3c0f013e7f38b (DEFAULT_TABLE table=test-workload [id=7bdc63e81a8f42aca69adc75753bc653]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260810 22:04:39.253268 30572 tablet_service.cc:1467] Processing CreateTablet for tablet 793f84134c904805bcf80ae4d941d6d4 (DEFAULT_TABLE table=test-workload [id=7bdc63e81a8f42aca69adc75753bc653]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260810 22:04:39.257231 31095 tablet_bootstrap.cc:492] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e: Bootstrap starting.
I20260810 22:04:39.257740 30574 tablet_service.cc:1467] Processing CreateTablet for tablet 21df603d01494efd960235764bf9e19b (DEFAULT_TABLE table=test-workload [id=7bdc63e81a8f42aca69adc75753bc653]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260810 22:04:39.258160 30567 tablet_service.cc:1467] Processing CreateTablet for tablet acf4ac19b8ea418aa7827d0c3df14ae0 (DEFAULT_TABLE table=test-workload [id=7bdc63e81a8f42aca69adc75753bc653]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260810 22:04:39.259253 30569 tablet_service.cc:1467] Processing CreateTablet for tablet f969babe0c5c4ae0a8a8e7162d361173 (DEFAULT_TABLE table=test-workload [id=7bdc63e81a8f42aca69adc75753bc653]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260810 22:04:39.271631 31092 tablet_bootstrap.cc:654] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:39.274893 31093 tablet_bootstrap.cc:492] T f969babe0c5c4ae0a8a8e7162d361173 P bd15e90d358641be9ce4e5563f57de30: Bootstrap starting.
I20260810 22:04:39.276672 31092 log.cc:824] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:39.297425 31092 tablet_bootstrap.cc:492] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e: No bootstrap required, opened a new log
I20260810 22:04:39.298101 31092 ts_tablet_manager.cc:1397] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e: Time spent bootstrapping tablet: real 0.045s user 0.016s sys 0.006s
I20260810 22:04:39.299678 31095 tablet_bootstrap.cc:654] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:39.311069 31099 tablet_bootstrap.cc:492] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30: Bootstrap starting.
I20260810 22:04:39.315285 31093 tablet_bootstrap.cc:654] T f969babe0c5c4ae0a8a8e7162d361173 P bd15e90d358641be9ce4e5563f57de30: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:39.322047 31091 tablet_bootstrap.cc:492] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30: No bootstrap required, opened a new log
I20260810 22:04:39.322621 31091 ts_tablet_manager.cc:1397] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30: Time spent bootstrapping tablet: real 0.102s user 0.021s sys 0.037s
I20260810 22:04:39.329790 31097 tablet_bootstrap.cc:492] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30: Bootstrap starting.
I20260810 22:04:39.332842 31099 tablet_bootstrap.cc:654] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:39.339994 31100 tablet_bootstrap.cc:492] T 21df603d01494efd960235764bf9e19b P 219a405626974999a6bdaa700b03120e: Bootstrap starting.
I20260810 22:04:39.346223 31095 tablet_bootstrap.cc:492] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e: No bootstrap required, opened a new log
I20260810 22:04:39.346662 31095 ts_tablet_manager.cc:1397] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e: Time spent bootstrapping tablet: real 0.091s user 0.013s sys 0.011s
I20260810 22:04:39.352692 31100 tablet_bootstrap.cc:654] T 21df603d01494efd960235764bf9e19b P 219a405626974999a6bdaa700b03120e: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:39.353823 31105 tablet_bootstrap.cc:492] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e: Bootstrap starting.
I20260810 22:04:39.371809 31102 tablet_bootstrap.cc:492] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e: Bootstrap starting.
I20260810 22:04:39.375160 31093 tablet_bootstrap.cc:492] T f969babe0c5c4ae0a8a8e7162d361173 P bd15e90d358641be9ce4e5563f57de30: No bootstrap required, opened a new log
I20260810 22:04:39.375607 31093 ts_tablet_manager.cc:1397] T f969babe0c5c4ae0a8a8e7162d361173 P bd15e90d358641be9ce4e5563f57de30: Time spent bootstrapping tablet: real 0.101s user 0.015s sys 0.000s
I20260810 22:04:39.388386 31106 tablet_bootstrap.cc:492] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30: Bootstrap starting.
I20260810 22:04:39.390791 31109 tablet_bootstrap.cc:492] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: Bootstrap starting.
I20260810 22:04:39.391650 31102 tablet_bootstrap.cc:654] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:39.395066 31100 tablet_bootstrap.cc:492] T 21df603d01494efd960235764bf9e19b P 219a405626974999a6bdaa700b03120e: No bootstrap required, opened a new log
I20260810 22:04:39.395498 31100 ts_tablet_manager.cc:1397] T 21df603d01494efd960235764bf9e19b P 219a405626974999a6bdaa700b03120e: Time spent bootstrapping tablet: real 0.056s user 0.014s sys 0.015s
I20260810 22:04:39.402634 31092 raft_consensus.cc:348] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.405294 31106 tablet_bootstrap.cc:654] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:39.406093 31092 raft_consensus.cc:374] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:39.406477 31092 raft_consensus.cc:729] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219a405626974999a6bdaa700b03120e, State: Initialized, Role: FOLLOWER
I20260810 22:04:39.403471 31105 tablet_bootstrap.cc:654] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:39.404104 31100 raft_consensus.cc:348] T 21df603d01494efd960235764bf9e19b P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } }
I20260810 22:04:39.407588 31100 raft_consensus.cc:374] T 21df603d01494efd960235764bf9e19b P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:39.407941 31100 raft_consensus.cc:729] T 21df603d01494efd960235764bf9e19b P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219a405626974999a6bdaa700b03120e, State: Initialized, Role: FOLLOWER
I20260810 22:04:39.408953 31100 consensus_queue.cc:260] T 21df603d01494efd960235764bf9e19b P 219a405626974999a6bdaa700b03120e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } }
I20260810 22:04:39.405195 31095 raft_consensus.cc:348] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.411566 31095 raft_consensus.cc:374] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:39.411900 31095 raft_consensus.cc:729] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219a405626974999a6bdaa700b03120e, State: Initialized, Role: FOLLOWER
I20260810 22:04:39.412714 31109 tablet_bootstrap.cc:654] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:39.412643 31095 consensus_queue.cc:260] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.399174 31113 tablet_bootstrap.cc:492] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: Bootstrap starting.
I20260810 22:04:39.402462 31110 tablet_bootstrap.cc:492] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8: Bootstrap starting.
I20260810 22:04:39.427587 31110 tablet_bootstrap.cc:654] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:39.415575 31109 log.cc:824] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:39.436182 31099 tablet_bootstrap.cc:492] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30: No bootstrap required, opened a new log
I20260810 22:04:39.436761 31099 ts_tablet_manager.cc:1397] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30: Time spent bootstrapping tablet: real 0.126s user 0.012s sys 0.004s
I20260810 22:04:39.415174 31092 consensus_queue.cc:260] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.439673 31092 ts_tablet_manager.cc:1428] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e: Time spent starting tablet: real 0.141s user 0.025s sys 0.004s
I20260810 22:04:39.440646 31118 raft_consensus.cc:482] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:39.441452 31095 ts_tablet_manager.cc:1428] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e: Time spent starting tablet: real 0.095s user 0.016s sys 0.010s
I20260810 22:04:39.447021 31097 tablet_bootstrap.cc:654] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:39.441291 31118 raft_consensus.cc:504] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.453465 31116 raft_consensus.cc:482] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:39.453909 31116 raft_consensus.cc:504] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.457267 31109 tablet_bootstrap.cc:492] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: No bootstrap required, opened a new log
I20260810 22:04:39.458061 31109 ts_tablet_manager.cc:1397] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: Time spent bootstrapping tablet: real 0.068s user 0.014s sys 0.004s
I20260810 22:04:39.458940 31111 tablet_bootstrap.cc:492] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8: Bootstrap starting.
I20260810 22:04:39.459556 31118 leader_election.cc:288] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815), bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619)
I20260810 22:04:39.460914 31113 tablet_bootstrap.cc:654] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:39.463658 31110 tablet_bootstrap.cc:492] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8: No bootstrap required, opened a new log
I20260810 22:04:39.465633 31110 ts_tablet_manager.cc:1397] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8: Time spent bootstrapping tablet: real 0.063s user 0.011s sys 0.004s
I20260810 22:04:39.467330 31116 leader_election.cc:288] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815), bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619)
I20260810 22:04:39.486241 31105 tablet_bootstrap.cc:492] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e: No bootstrap required, opened a new log
I20260810 22:04:39.486821 31105 ts_tablet_manager.cc:1397] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e: Time spent bootstrapping tablet: real 0.150s user 0.017s sys 0.004s
I20260810 22:04:39.490075 31105 raft_consensus.cc:348] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.490844 31105 raft_consensus.cc:374] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:39.491128 31105 raft_consensus.cc:729] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219a405626974999a6bdaa700b03120e, State: Initialized, Role: FOLLOWER
I20260810 22:04:39.490808 30611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf4ac19b8ea418aa7827d0c3df14ae0" candidate_uuid: "219a405626974999a6bdaa700b03120e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842b26ed57e14f5784632195da785fc8" is_pre_election: true
I20260810 22:04:39.491241 30610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71556c430684bc889e3c0f013e7f38b" candidate_uuid: "219a405626974999a6bdaa700b03120e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842b26ed57e14f5784632195da785fc8" is_pre_election: true
I20260810 22:04:39.491905 31105 consensus_queue.cc:260] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.492545 31113 tablet_bootstrap.cc:492] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: No bootstrap required, opened a new log
I20260810 22:04:39.493028 31113 ts_tablet_manager.cc:1397] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: Time spent bootstrapping tablet: real 0.094s user 0.004s sys 0.009s
W20260810 22:04:39.493453 30725 leader_election.cc:341] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:04:39.494966 31127 tablet_bootstrap.cc:492] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: Bootstrap starting.
W20260810 22:04:39.494203 30725 leader_election.cc:341] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:04:39.498863 31105 ts_tablet_manager.cc:1428] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260810 22:04:39.508157 31116 raft_consensus.cc:482] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:39.508635 31116 raft_consensus.cc:504] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.510025 31110 raft_consensus.cc:348] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.510643 31116 leader_election.cc:288] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815), bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619)
I20260810 22:04:39.511238 31106 tablet_bootstrap.cc:492] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30: No bootstrap required, opened a new log
I20260810 22:04:39.511364 30610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f84134c904805bcf80ae4d941d6d4" candidate_uuid: "219a405626974999a6bdaa700b03120e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842b26ed57e14f5784632195da785fc8" is_pre_election: true
I20260810 22:04:39.511796 31106 ts_tablet_manager.cc:1397] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30: Time spent bootstrapping tablet: real 0.124s user 0.006s sys 0.008s
I20260810 22:04:39.511118 31109 raft_consensus.cc:348] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.512480 31109 raft_consensus.cc:374] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:39.512840 31109 raft_consensus.cc:729] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842b26ed57e14f5784632195da785fc8, State: Initialized, Role: FOLLOWER
I20260810 22:04:39.513290 31110 raft_consensus.cc:374] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260810 22:04:39.513513 30725 leader_election.cc:341] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:04:39.514050 31110 raft_consensus.cc:729] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842b26ed57e14f5784632195da785fc8, State: Initialized, Role: FOLLOWER
I20260810 22:04:39.515686 31109 consensus_queue.cc:260] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.516515 30993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71556c430684bc889e3c0f013e7f38b" candidate_uuid: "219a405626974999a6bdaa700b03120e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd15e90d358641be9ce4e5563f57de30" is_pre_election: true
I20260810 22:04:39.516846 30994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf4ac19b8ea418aa7827d0c3df14ae0" candidate_uuid: "219a405626974999a6bdaa700b03120e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd15e90d358641be9ce4e5563f57de30" is_pre_election: true
I20260810 22:04:39.516974 31113 raft_consensus.cc:348] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.518514 30992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f84134c904805bcf80ae4d941d6d4" candidate_uuid: "219a405626974999a6bdaa700b03120e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd15e90d358641be9ce4e5563f57de30" is_pre_election: true
I20260810 22:04:39.519217 31113 raft_consensus.cc:374] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:39.519547 31113 raft_consensus.cc:729] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842b26ed57e14f5784632195da785fc8, State: Initialized, Role: FOLLOWER
I20260810 22:04:39.519706 31102 tablet_bootstrap.cc:492] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e: No bootstrap required, opened a new log
I20260810 22:04:39.520152 31102 ts_tablet_manager.cc:1397] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e: Time spent bootstrapping tablet: real 0.149s user 0.025s sys 0.059s
W20260810 22:04:39.520572 30726 leader_election.cc:341] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:04:39.520996 30726 leader_election.cc:302] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 219a405626974999a6bdaa700b03120e; no voters: 842b26ed57e14f5784632195da785fc8, bd15e90d358641be9ce4e5563f57de30
I20260810 22:04:39.520469 31110 consensus_queue.cc:260] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.522395 31100 ts_tablet_manager.cc:1428] T 21df603d01494efd960235764bf9e19b P 219a405626974999a6bdaa700b03120e: Time spent starting tablet: real 0.127s user 0.013s sys 0.007s
W20260810 22:04:39.522454 30726 leader_election.cc:341] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:04:39.522955 30726 leader_election.cc:302] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 219a405626974999a6bdaa700b03120e; no voters: 842b26ed57e14f5784632195da785fc8, bd15e90d358641be9ce4e5563f57de30
I20260810 22:04:39.523801 31118 raft_consensus.cc:2726] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 22:04:39.524384 31118 raft_consensus.cc:2726] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 22:04:39.523159 31113 consensus_queue.cc:260] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
W20260810 22:04:39.525436 30726 leader_election.cc:341] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:04:39.525905 30726 leader_election.cc:302] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 219a405626974999a6bdaa700b03120e; no voters: 842b26ed57e14f5784632195da785fc8, bd15e90d358641be9ce4e5563f57de30
I20260810 22:04:39.526631 31116 raft_consensus.cc:2726] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 22:04:39.528766 31099 raft_consensus.cc:348] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.529175 31102 raft_consensus.cc:348] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.529762 31099 raft_consensus.cc:374] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:39.530053 31102 raft_consensus.cc:374] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:39.530040 31099 raft_consensus.cc:729] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd15e90d358641be9ce4e5563f57de30, State: Initialized, Role: FOLLOWER
I20260810 22:04:39.530339 31102 raft_consensus.cc:729] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219a405626974999a6bdaa700b03120e, State: Initialized, Role: FOLLOWER
I20260810 22:04:39.531129 31102 consensus_queue.cc:260] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.531172 31099 consensus_queue.cc:260] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.527954 31091 raft_consensus.cc:348] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.534869 31091 raft_consensus.cc:374] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:39.535145 31091 raft_consensus.cc:729] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd15e90d358641be9ce4e5563f57de30, State: Initialized, Role: FOLLOWER
I20260810 22:04:39.535377 31109 ts_tablet_manager.cc:1428] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: Time spent starting tablet: real 0.077s user 0.021s sys 0.004s
I20260810 22:04:39.535923 31091 consensus_queue.cc:260] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.537375 31110 ts_tablet_manager.cc:1428] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8: Time spent starting tablet: real 0.069s user 0.013s sys 0.007s
I20260810 22:04:39.539324 31054 heartbeater.cc:503] Master 127.27.17.190:33411 was elected leader, sending a full tablet report...
I20260810 22:04:39.540777 31127 tablet_bootstrap.cc:654] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:39.541569 31111 tablet_bootstrap.cc:654] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:39.541642 31102 ts_tablet_manager.cc:1428] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e: Time spent starting tablet: real 0.021s user 0.009s sys 0.000s
I20260810 22:04:39.542505 31113 ts_tablet_manager.cc:1428] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: Time spent starting tablet: real 0.049s user 0.017s sys 0.009s
I20260810 22:04:39.551476 31093 raft_consensus.cc:348] T f969babe0c5c4ae0a8a8e7162d361173 P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.552071 31093 raft_consensus.cc:374] T f969babe0c5c4ae0a8a8e7162d361173 P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:39.552428 31093 raft_consensus.cc:729] T f969babe0c5c4ae0a8a8e7162d361173 P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd15e90d358641be9ce4e5563f57de30, State: Initialized, Role: FOLLOWER
I20260810 22:04:39.553257 31093 consensus_queue.cc:260] T f969babe0c5c4ae0a8a8e7162d361173 P bd15e90d358641be9ce4e5563f57de30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.558502 31093 ts_tablet_manager.cc:1428] T f969babe0c5c4ae0a8a8e7162d361173 P bd15e90d358641be9ce4e5563f57de30: Time spent starting tablet: real 0.183s user 0.025s sys 0.000s
I20260810 22:04:39.564620 31097 tablet_bootstrap.cc:492] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30: No bootstrap required, opened a new log
W20260810 22:04:39.564867 30873 tablet.cc:2369] T 21df603d01494efd960235764bf9e19b P 219a405626974999a6bdaa700b03120e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:04:39.565076 31097 ts_tablet_manager.cc:1397] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30: Time spent bootstrapping tablet: real 0.236s user 0.002s sys 0.012s
I20260810 22:04:39.570839 31111 tablet_bootstrap.cc:492] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8: No bootstrap required, opened a new log
I20260810 22:04:39.571278 31111 ts_tablet_manager.cc:1397] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8: Time spent bootstrapping tablet: real 0.113s user 0.007s sys 0.012s
I20260810 22:04:39.571657 31091 ts_tablet_manager.cc:1428] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30: Time spent starting tablet: real 0.249s user 0.022s sys 0.001s
I20260810 22:04:39.571842 31127 tablet_bootstrap.cc:492] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: No bootstrap required, opened a new log
I20260810 22:04:39.572227 31127 ts_tablet_manager.cc:1397] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: Time spent bootstrapping tablet: real 0.078s user 0.010s sys 0.014s
I20260810 22:04:39.572111 31106 raft_consensus.cc:348] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.573021 31106 raft_consensus.cc:374] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:39.573879 31111 raft_consensus.cc:348] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.574651 31111 raft_consensus.cc:374] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:39.573300 31106 raft_consensus.cc:729] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd15e90d358641be9ce4e5563f57de30, State: Initialized, Role: FOLLOWER
I20260810 22:04:39.574940 31111 raft_consensus.cc:729] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842b26ed57e14f5784632195da785fc8, State: Initialized, Role: FOLLOWER
I20260810 22:04:39.575398 31106 consensus_queue.cc:260] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.575675 31111 consensus_queue.cc:260] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.576054 31127 raft_consensus.cc:348] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } }
I20260810 22:04:39.576756 31127 raft_consensus.cc:374] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:39.577028 31127 raft_consensus.cc:729] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842b26ed57e14f5784632195da785fc8, State: Initialized, Role: FOLLOWER
I20260810 22:04:39.578223 31106 ts_tablet_manager.cc:1428] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30: Time spent starting tablet: real 0.066s user 0.010s sys 0.000s
I20260810 22:04:39.578233 31111 ts_tablet_manager.cc:1428] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260810 22:04:39.577777 31127 consensus_queue.cc:260] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } }
I20260810 22:04:39.579893 31127 ts_tablet_manager.cc:1428] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260810 22:04:39.580631 31097 raft_consensus.cc:348] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } }
I20260810 22:04:39.581543 31097 raft_consensus.cc:374] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:39.581856 31097 raft_consensus.cc:729] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd15e90d358641be9ce4e5563f57de30, State: Initialized, Role: FOLLOWER
I20260810 22:04:39.582156 31099 ts_tablet_manager.cc:1428] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30: Time spent starting tablet: real 0.144s user 0.005s sys 0.015s
I20260810 22:04:39.582717 31097 consensus_queue.cc:260] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } }
I20260810 22:04:39.584381 31097 ts_tablet_manager.cc:1428] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30: Time spent starting tablet: real 0.019s user 0.003s sys 0.003s
I20260810 22:04:39.648837 31116 raft_consensus.cc:482] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:39.649250 31116 raft_consensus.cc:504] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.650714 31116 leader_election.cc:288] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815), bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619)
I20260810 22:04:39.651705 30610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f969babe0c5c4ae0a8a8e7162d361173" candidate_uuid: "219a405626974999a6bdaa700b03120e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842b26ed57e14f5784632195da785fc8" is_pre_election: true
I20260810 22:04:39.651842 30992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f969babe0c5c4ae0a8a8e7162d361173" candidate_uuid: "219a405626974999a6bdaa700b03120e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd15e90d358641be9ce4e5563f57de30" is_pre_election: true
I20260810 22:04:39.652379 30610 raft_consensus.cc:2454] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 219a405626974999a6bdaa700b03120e in term 0.
I20260810 22:04:39.652585 30992 raft_consensus.cc:2454] T f969babe0c5c4ae0a8a8e7162d361173 P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 219a405626974999a6bdaa700b03120e in term 0.
I20260810 22:04:39.653450 30725 leader_election.cc:302] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219a405626974999a6bdaa700b03120e, 842b26ed57e14f5784632195da785fc8; no voters:
I20260810 22:04:39.654218 31116 raft_consensus.cc:2781] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:39.654480 31116 raft_consensus.cc:482] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:39.654748 31116 raft_consensus.cc:3037] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:39.659570 31116 raft_consensus.cc:504] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.661106 31116 leader_election.cc:288] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e [CANDIDATE]: Term 1 election: Requested vote from peers 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815), bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619)
W20260810 22:04:39.661610 30683 tablet.cc:2369] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:04:39.662215 30610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f969babe0c5c4ae0a8a8e7162d361173" candidate_uuid: "219a405626974999a6bdaa700b03120e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842b26ed57e14f5784632195da785fc8"
I20260810 22:04:39.662335 30992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f969babe0c5c4ae0a8a8e7162d361173" candidate_uuid: "219a405626974999a6bdaa700b03120e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd15e90d358641be9ce4e5563f57de30"
I20260810 22:04:39.662762 30610 raft_consensus.cc:3037] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:39.662845 30992 raft_consensus.cc:3037] T f969babe0c5c4ae0a8a8e7162d361173 P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:39.667419 30610 raft_consensus.cc:2454] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 219a405626974999a6bdaa700b03120e in term 1.
I20260810 22:04:39.668318 30725 leader_election.cc:302] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219a405626974999a6bdaa700b03120e, 842b26ed57e14f5784632195da785fc8; no voters:
I20260810 22:04:39.668440 30992 raft_consensus.cc:2454] T f969babe0c5c4ae0a8a8e7162d361173 P bd15e90d358641be9ce4e5563f57de30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 219a405626974999a6bdaa700b03120e in term 1.
I20260810 22:04:39.669005 31116 raft_consensus.cc:2781] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:39.669482 31116 raft_consensus.cc:686] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e [term 1 LEADER]: Becoming Leader. State: Replica: 219a405626974999a6bdaa700b03120e, State: Running, Role: LEADER
I20260810 22:04:39.670258 31116 consensus_queue.cc:237] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.677906 30392 catalog_manager.cc:5495] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e reported cstate change: term changed from 0 to 1, leader changed from <none> to 219a405626974999a6bdaa700b03120e (127.27.17.130). New cstate: current_term: 1 leader_uuid: "219a405626974999a6bdaa700b03120e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } }
I20260810 22:04:39.718704 31136 raft_consensus.cc:482] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:39.719201 31136 raft_consensus.cc:504] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } }
I20260810 22:04:39.721695 31136 leader_election.cc:288] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815), 219a405626974999a6bdaa700b03120e (127.27.17.130:36727)
I20260810 22:04:39.739670 30812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21df603d01494efd960235764bf9e19b" candidate_uuid: "bd15e90d358641be9ce4e5563f57de30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219a405626974999a6bdaa700b03120e" is_pre_election: true
I20260810 22:04:39.740274 30610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21df603d01494efd960235764bf9e19b" candidate_uuid: "bd15e90d358641be9ce4e5563f57de30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842b26ed57e14f5784632195da785fc8" is_pre_election: true
I20260810 22:04:39.740537 30812 raft_consensus.cc:2454] T 21df603d01494efd960235764bf9e19b P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd15e90d358641be9ce4e5563f57de30 in term 0.
I20260810 22:04:39.740880 30610 raft_consensus.cc:2454] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd15e90d358641be9ce4e5563f57de30 in term 0.
I20260810 22:04:39.741890 30912 leader_election.cc:302] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219a405626974999a6bdaa700b03120e, bd15e90d358641be9ce4e5563f57de30; no voters:
I20260810 22:04:39.742677 31136 raft_consensus.cc:2781] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:39.742969 31136 raft_consensus.cc:482] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:39.743232 31136 raft_consensus.cc:3037] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:39.747555 31136 raft_consensus.cc:504] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } }
I20260810 22:04:39.748944 31136 leader_election.cc:288] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 1 election: Requested vote from peers 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815), 219a405626974999a6bdaa700b03120e (127.27.17.130:36727)
I20260810 22:04:39.750056 30812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21df603d01494efd960235764bf9e19b" candidate_uuid: "bd15e90d358641be9ce4e5563f57de30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219a405626974999a6bdaa700b03120e"
I20260810 22:04:39.750275 30610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21df603d01494efd960235764bf9e19b" candidate_uuid: "bd15e90d358641be9ce4e5563f57de30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842b26ed57e14f5784632195da785fc8"
I20260810 22:04:39.750553 30812 raft_consensus.cc:3037] T 21df603d01494efd960235764bf9e19b P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:39.750777 30610 raft_consensus.cc:3037] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Advancing to term 1
W20260810 22:04:39.752293 31055 tablet.cc:2369] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:04:39.755143 30610 raft_consensus.cc:2454] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd15e90d358641be9ce4e5563f57de30 in term 1.
I20260810 22:04:39.756273 30914 leader_election.cc:302] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 842b26ed57e14f5784632195da785fc8, bd15e90d358641be9ce4e5563f57de30; no voters:
I20260810 22:04:39.757170 31136 raft_consensus.cc:2781] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:39.757630 31136 raft_consensus.cc:686] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [term 1 LEADER]: Becoming Leader. State: Replica: bd15e90d358641be9ce4e5563f57de30, State: Running, Role: LEADER
I20260810 22:04:39.757298 30812 raft_consensus.cc:2454] T 21df603d01494efd960235764bf9e19b P 219a405626974999a6bdaa700b03120e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd15e90d358641be9ce4e5563f57de30 in term 1.
I20260810 22:04:39.758380 31136 consensus_queue.cc:237] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } }
I20260810 22:04:39.767949 30392 catalog_manager.cc:5495] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd15e90d358641be9ce4e5563f57de30 (127.27.17.131). New cstate: current_term: 1 leader_uuid: "bd15e90d358641be9ce4e5563f57de30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } }
I20260810 22:04:39.797896 31134 raft_consensus.cc:482] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:39.798350 31134 raft_consensus.cc:504] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.800788 31134 leader_election.cc:288] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 219a405626974999a6bdaa700b03120e (127.27.17.130:36727), bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619)
I20260810 22:04:39.812690 30812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f84134c904805bcf80ae4d941d6d4" candidate_uuid: "842b26ed57e14f5784632195da785fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219a405626974999a6bdaa700b03120e" is_pre_election: true
I20260810 22:04:39.813239 30812 raft_consensus.cc:2454] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 842b26ed57e14f5784632195da785fc8 in term 0.
I20260810 22:04:39.813689 31136 raft_consensus.cc:482] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:39.814378 30992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f84134c904805bcf80ae4d941d6d4" candidate_uuid: "842b26ed57e14f5784632195da785fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd15e90d358641be9ce4e5563f57de30" is_pre_election: true
I20260810 22:04:39.814147 31136 raft_consensus.cc:504] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.814499 30506 leader_election.cc:302] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219a405626974999a6bdaa700b03120e, 842b26ed57e14f5784632195da785fc8; no voters:
I20260810 22:04:39.815012 30992 raft_consensus.cc:2454] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 842b26ed57e14f5784632195da785fc8 in term 0.
I20260810 22:04:39.815392 31134 raft_consensus.cc:2781] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:39.815757 31134 raft_consensus.cc:482] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:39.816107 31134 raft_consensus.cc:3037] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:39.816603 31136 leader_election.cc:288] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815), 219a405626974999a6bdaa700b03120e (127.27.17.130:36727)
I20260810 22:04:39.817363 30610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf4ac19b8ea418aa7827d0c3df14ae0" candidate_uuid: "bd15e90d358641be9ce4e5563f57de30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842b26ed57e14f5784632195da785fc8" is_pre_election: true
I20260810 22:04:39.817667 30812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf4ac19b8ea418aa7827d0c3df14ae0" candidate_uuid: "bd15e90d358641be9ce4e5563f57de30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219a405626974999a6bdaa700b03120e" is_pre_election: true
I20260810 22:04:39.818028 30610 raft_consensus.cc:2454] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd15e90d358641be9ce4e5563f57de30 in term 0.
I20260810 22:04:39.818277 30812 raft_consensus.cc:2454] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd15e90d358641be9ce4e5563f57de30 in term 0.
I20260810 22:04:39.819205 30912 leader_election.cc:302] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219a405626974999a6bdaa700b03120e, bd15e90d358641be9ce4e5563f57de30; no voters:
I20260810 22:04:39.820019 31136 raft_consensus.cc:2781] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:39.820348 31136 raft_consensus.cc:482] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:39.820632 31136 raft_consensus.cc:3037] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:39.821863 31134 raft_consensus.cc:504] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.823237 31134 leader_election.cc:288] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [CANDIDATE]: Term 1 election: Requested vote from peers 219a405626974999a6bdaa700b03120e (127.27.17.130:36727), bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619)
I20260810 22:04:39.823990 30812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f84134c904805bcf80ae4d941d6d4" candidate_uuid: "842b26ed57e14f5784632195da785fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219a405626974999a6bdaa700b03120e"
I20260810 22:04:39.824528 30812 raft_consensus.cc:3037] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:39.824589 30992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f84134c904805bcf80ae4d941d6d4" candidate_uuid: "842b26ed57e14f5784632195da785fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd15e90d358641be9ce4e5563f57de30"
I20260810 22:04:39.825124 30992 raft_consensus.cc:3037] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:39.825510 31136 raft_consensus.cc:504] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.827281 31136 leader_election.cc:288] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 1 election: Requested vote from peers 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815), 219a405626974999a6bdaa700b03120e (127.27.17.130:36727)
I20260810 22:04:39.828466 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf4ac19b8ea418aa7827d0c3df14ae0" candidate_uuid: "bd15e90d358641be9ce4e5563f57de30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219a405626974999a6bdaa700b03120e"
I20260810 22:04:39.829089 30810 raft_consensus.cc:3037] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:39.829048 30610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf4ac19b8ea418aa7827d0c3df14ae0" candidate_uuid: "bd15e90d358641be9ce4e5563f57de30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842b26ed57e14f5784632195da785fc8"
I20260810 22:04:39.829583 30812 raft_consensus.cc:2454] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842b26ed57e14f5784632195da785fc8 in term 1.
I20260810 22:04:39.829757 30610 raft_consensus.cc:3037] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:39.830562 30506 leader_election.cc:302] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219a405626974999a6bdaa700b03120e, 842b26ed57e14f5784632195da785fc8; no voters:
I20260810 22:04:39.831303 31134 raft_consensus.cc:2781] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:39.831429 30992 raft_consensus.cc:2454] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842b26ed57e14f5784632195da785fc8 in term 1.
I20260810 22:04:39.831758 31134 raft_consensus.cc:686] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [term 1 LEADER]: Becoming Leader. State: Replica: 842b26ed57e14f5784632195da785fc8, State: Running, Role: LEADER
I20260810 22:04:39.832705 31134 consensus_queue.cc:237] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.835525 30810 raft_consensus.cc:2454] T acf4ac19b8ea418aa7827d0c3df14ae0 P 219a405626974999a6bdaa700b03120e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd15e90d358641be9ce4e5563f57de30 in term 1.
I20260810 22:04:39.836155 30610 raft_consensus.cc:2454] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd15e90d358641be9ce4e5563f57de30 in term 1.
I20260810 22:04:39.836402 30912 leader_election.cc:302] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219a405626974999a6bdaa700b03120e, bd15e90d358641be9ce4e5563f57de30; no voters:
I20260810 22:04:39.837306 31136 raft_consensus.cc:2781] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:39.837795 31136 raft_consensus.cc:686] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 [term 1 LEADER]: Becoming Leader. State: Replica: bd15e90d358641be9ce4e5563f57de30, State: Running, Role: LEADER
I20260810 22:04:39.840627 31136 consensus_queue.cc:237] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.843065 30392 catalog_manager.cc:5495] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 842b26ed57e14f5784632195da785fc8 (127.27.17.129). New cstate: current_term: 1 leader_uuid: "842b26ed57e14f5784632195da785fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } }
I20260810 22:04:39.847388 30391 catalog_manager.cc:5495] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd15e90d358641be9ce4e5563f57de30 (127.27.17.131). New cstate: current_term: 1 leader_uuid: "bd15e90d358641be9ce4e5563f57de30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } }
I20260810 22:04:39.930148 31134 raft_consensus.cc:482] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:39.930642 31134 raft_consensus.cc:504] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.932314 31134 leader_election.cc:288] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 219a405626974999a6bdaa700b03120e (127.27.17.130:36727), bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619)
I20260810 22:04:39.933284 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71556c430684bc889e3c0f013e7f38b" candidate_uuid: "842b26ed57e14f5784632195da785fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219a405626974999a6bdaa700b03120e" is_pre_election: true
I20260810 22:04:39.933501 30992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71556c430684bc889e3c0f013e7f38b" candidate_uuid: "842b26ed57e14f5784632195da785fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd15e90d358641be9ce4e5563f57de30" is_pre_election: true
I20260810 22:04:39.934046 30810 raft_consensus.cc:2454] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 842b26ed57e14f5784632195da785fc8 in term 0.
I20260810 22:04:39.934154 30992 raft_consensus.cc:2454] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 842b26ed57e14f5784632195da785fc8 in term 0.
I20260810 22:04:39.935156 30506 leader_election.cc:302] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219a405626974999a6bdaa700b03120e, 842b26ed57e14f5784632195da785fc8; no voters:
I20260810 22:04:39.935833 31134 raft_consensus.cc:2781] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:39.936163 31134 raft_consensus.cc:482] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:39.936475 31134 raft_consensus.cc:3037] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:39.940886 31134 raft_consensus.cc:504] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.942540 31134 leader_election.cc:288] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [CANDIDATE]: Term 1 election: Requested vote from peers 219a405626974999a6bdaa700b03120e (127.27.17.130:36727), bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619)
I20260810 22:04:39.943480 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71556c430684bc889e3c0f013e7f38b" candidate_uuid: "842b26ed57e14f5784632195da785fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219a405626974999a6bdaa700b03120e"
I20260810 22:04:39.943641 30992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71556c430684bc889e3c0f013e7f38b" candidate_uuid: "842b26ed57e14f5784632195da785fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd15e90d358641be9ce4e5563f57de30"
I20260810 22:04:39.943943 30810 raft_consensus.cc:3037] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:39.944172 30992 raft_consensus.cc:3037] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:39.948500 30992 raft_consensus.cc:2454] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842b26ed57e14f5784632195da785fc8 in term 1.
I20260810 22:04:39.948562 30810 raft_consensus.cc:2454] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842b26ed57e14f5784632195da785fc8 in term 1.
I20260810 22:04:39.949492 30506 leader_election.cc:302] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219a405626974999a6bdaa700b03120e, 842b26ed57e14f5784632195da785fc8; no voters:
I20260810 22:04:39.950165 31134 raft_consensus.cc:2781] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:39.950609 31134 raft_consensus.cc:686] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [term 1 LEADER]: Becoming Leader. State: Replica: 842b26ed57e14f5784632195da785fc8, State: Running, Role: LEADER
I20260810 22:04:39.951229 31134 consensus_queue.cc:237] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:39.960839 30391 catalog_manager.cc:5495] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 842b26ed57e14f5784632195da785fc8 (127.27.17.129). New cstate: current_term: 1 leader_uuid: "842b26ed57e14f5784632195da785fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } }
I20260810 22:04:40.125686 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 30475
I20260810 22:04:40.159307 31116 consensus_queue.cc:1035] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 22:04:40.174610 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.129:37815
--local_ip_for_outbound_sockets=127.27.17.129
--tserver_master_addrs=127.27.17.190:33411
--webserver_port=37987
--webserver_interface=127.27.17.129
--env_inject_eio_globs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-1/**
--env_inject_eio=1.0
--crash_on_eio=false
--builtin_ntp_servers=127.27.17.148:42125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20260810 22:04:40.181136 31136 consensus_queue.cc:1035] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 22:04:40.181659 30725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.129:37815: connect: Connection refused (error 111)
W20260810 22:04:40.187762 30725 consensus_peers.cc:487] T f969babe0c5c4ae0a8a8e7162d361173 P 219a405626974999a6bdaa700b03120e -> Peer 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Couldn't send request to peer 842b26ed57e14f5784632195da785fc8. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 22:04:40.192473 30914 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.129:37815: connect: Connection refused (error 111)
W20260810 22:04:40.196610 30914 consensus_peers.cc:487] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 -> Peer 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Couldn't send request to peer 842b26ed57e14f5784632195da785fc8. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:40.257266 31137 consensus_queue.cc:1035] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 22:04:40.270705 30914 consensus_peers.cc:487] T acf4ac19b8ea418aa7827d0c3df14ae0 P bd15e90d358641be9ce4e5563f57de30 -> Peer 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Couldn't send request to peer 842b26ed57e14f5784632195da785fc8. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 22:04:40.509992 31169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:40.510537 31169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:40.511162 31169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:40.543059 31169 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260810 22:04:40.543502 31169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:40.544224 31169 flags.cc:425] Enabled experimental flag: --crash_on_eio=false
W20260810 22:04:40.544796 31169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.129
I20260810 22:04:40.582208 31169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:42125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.129:37815
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.27.17.129
--webserver_port=37987
--tserver_master_addrs=127.27.17.190:33411
--env_inject_eio=1
--env_inject_eio_globs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-1/**
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.129
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:40.583449 31169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260810 22:04:40.584326 31169 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260810 22:04:40.584465 31169 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260810 22:04:40.585541 31169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:40.604410 31186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:41.464918 31176 raft_consensus.cc:482] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:41.465355 31176 raft_consensus.cc:504] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:41.466984 31176 leader_election.cc:288] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815), 219a405626974999a6bdaa700b03120e (127.27.17.130:36727)
I20260810 22:04:41.468191 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71556c430684bc889e3c0f013e7f38b" candidate_uuid: "bd15e90d358641be9ce4e5563f57de30" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219a405626974999a6bdaa700b03120e" is_pre_election: true
I20260810 22:04:41.468832 30810 raft_consensus.cc:2454] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd15e90d358641be9ce4e5563f57de30 in term 1.
I20260810 22:04:41.470018 30912 leader_election.cc:302] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219a405626974999a6bdaa700b03120e, bd15e90d358641be9ce4e5563f57de30; no voters:
W20260810 22:04:41.470963 30914 leader_election.cc:334] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Network error: Client connection negotiation failed: client connection to 127.27.17.129:37815: connect: Connection refused (error 111)
I20260810 22:04:41.471112 31176 raft_consensus.cc:2781] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 22:04:41.471416 31176 raft_consensus.cc:482] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:41.471731 31176 raft_consensus.cc:3037] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:04:41.476099 31176 raft_consensus.cc:504] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:41.477646 31176 leader_election.cc:288] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 2 election: Requested vote from peers 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815), 219a405626974999a6bdaa700b03120e (127.27.17.130:36727)
I20260810 22:04:41.479179 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b71556c430684bc889e3c0f013e7f38b" candidate_uuid: "bd15e90d358641be9ce4e5563f57de30" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219a405626974999a6bdaa700b03120e"
I20260810 22:04:41.480017 30810 raft_consensus.cc:3037] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [term 1 FOLLOWER]: Advancing to term 2
W20260810 22:04:41.480662 30914 leader_election.cc:334] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Network error: Client connection negotiation failed: client connection to 127.27.17.129:37815: connect: Connection refused (error 111)
I20260810 22:04:41.489499 30810 raft_consensus.cc:2454] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd15e90d358641be9ce4e5563f57de30 in term 2.
I20260810 22:04:41.491066 30912 leader_election.cc:302] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 219a405626974999a6bdaa700b03120e, bd15e90d358641be9ce4e5563f57de30; no voters: 842b26ed57e14f5784632195da785fc8
I20260810 22:04:41.492138 31176 raft_consensus.cc:2781] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:04:41.493265 31176 raft_consensus.cc:686] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [term 2 LEADER]: Becoming Leader. State: Replica: bd15e90d358641be9ce4e5563f57de30, State: Running, Role: LEADER
I20260810 22:04:41.494035 31176 consensus_queue.cc:237] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:41.503299 30391 catalog_manager.cc:5495] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 reported cstate change: term changed from 1 to 2, leader changed from 842b26ed57e14f5784632195da785fc8 (127.27.17.129) to bd15e90d358641be9ce4e5563f57de30 (127.27.17.131). New cstate: current_term: 2 leader_uuid: "bd15e90d358641be9ce4e5563f57de30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } }
W20260810 22:04:40.604423 31190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:40.605052 31185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:41.834457 31189 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 22:04:41.834847 31169 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 22:04:41.839594 31169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:41.842698 31169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:41.844128 31169 hybrid_clock.cc:648] HybridClock initialized: now 1786399481844084 us; error 38 us; skew 500 ppm
I20260810 22:04:41.844991 31169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:41.851686 31169 webserver.cc:460] Webserver started at http://127.27.17.129:37987/ using document root <none> and password file <none>
I20260810 22:04:41.853479 31169 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:41.853760 31169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
E20260810 22:04:41.856689 31169 env_posix.cc:432] I/O error, context: INJECTED FAILURE
E20260810 22:04:41.861656 31169 env_posix.cc:432] I/O error, context: INJECTED FAILURE
E20260810 22:04:41.877087 31169 env_posix.cc:432] I/O error, context: INJECTED FAILURE
I20260810 22:04:41.889302 31169 fs_manager.cc:709] Time spent opening directory manager: real 0.029s user 0.022s sys 0.008s
I20260810 22:04:41.893618 31176 raft_consensus.cc:482] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:41.894145 31176 raft_consensus.cc:504] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
E20260810 22:04:41.895139 31169 log_block_manager.cc:2924] Not using report from /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-1/data: IO error: Data directory failed (error 5)
I20260810 22:04:41.895534 31212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:41.896080 31212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:41.896490 31176 leader_election.cc:288] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815), 219a405626974999a6bdaa700b03120e (127.27.17.130:36727)
I20260810 22:04:41.897243 31212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:41.897387 31216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:41.898468 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f84134c904805bcf80ae4d941d6d4" candidate_uuid: "bd15e90d358641be9ce4e5563f57de30" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219a405626974999a6bdaa700b03120e" is_pre_election: true
I20260810 22:04:41.899080 30810 raft_consensus.cc:2454] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd15e90d358641be9ce4e5563f57de30 in term 1.
I20260810 22:04:41.899616 31169 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.003s
I20260810 22:04:41.899951 30912 leader_election.cc:302] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219a405626974999a6bdaa700b03120e, bd15e90d358641be9ce4e5563f57de30; no voters:
I20260810 22:04:41.900019 31169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/wal
uuid: "842b26ed57e14f5784632195da785fc8"
format_stamp: "Formatted at 2026-08-10 22:04:34 on dist-test-slave-bw88"
I20260810 22:04:41.900700 31176 raft_consensus.cc:2781] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260810 22:04:41.900876 30914 leader_election.cc:334] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Network error: Client connection negotiation failed: client connection to 127.27.17.129:37815: connect: Connection refused (error 111)
I20260810 22:04:41.901049 31176 raft_consensus.cc:482] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:41.900803 31169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/wal
4 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:04:41.901496 31176 raft_consensus.cc:3037] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:04:41.906128 31176 raft_consensus.cc:504] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:41.907702 31176 leader_election.cc:288] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 2 election: Requested vote from peers 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815), 219a405626974999a6bdaa700b03120e (127.27.17.130:36727)
I20260810 22:04:41.908923 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "793f84134c904805bcf80ae4d941d6d4" candidate_uuid: "bd15e90d358641be9ce4e5563f57de30" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219a405626974999a6bdaa700b03120e"
I20260810 22:04:41.909444 30810 raft_consensus.cc:3037] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 1 FOLLOWER]: Advancing to term 2
W20260810 22:04:41.911676 30914 leader_election.cc:334] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Network error: Client connection negotiation failed: client connection to 127.27.17.129:37815: connect: Connection refused (error 111)
I20260810 22:04:41.914870 30810 raft_consensus.cc:2454] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd15e90d358641be9ce4e5563f57de30 in term 2.
I20260810 22:04:41.915848 30912 leader_election.cc:302] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 219a405626974999a6bdaa700b03120e, bd15e90d358641be9ce4e5563f57de30; no voters: 842b26ed57e14f5784632195da785fc8
I20260810 22:04:41.916564 31176 raft_consensus.cc:2781] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:04:41.917014 31176 raft_consensus.cc:686] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [term 2 LEADER]: Becoming Leader. State: Replica: bd15e90d358641be9ce4e5563f57de30, State: Running, Role: LEADER
I20260810 22:04:41.917826 31176 consensus_queue.cc:237] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:41.925956 30810 raft_consensus.cc:1261] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [term 2 FOLLOWER]: Refusing update from remote peer bd15e90d358641be9ce4e5563f57de30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260810 22:04:41.927299 31176 consensus_queue.cc:1035] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:04:41.926596 30391 catalog_manager.cc:5495] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 reported cstate change: term changed from 1 to 2, leader changed from 842b26ed57e14f5784632195da785fc8 (127.27.17.129) to bd15e90d358641be9ce4e5563f57de30 (127.27.17.131). New cstate: current_term: 2 leader_uuid: "bd15e90d358641be9ce4e5563f57de30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } }
W20260810 22:04:41.958562 30914 consensus_peers.cc:487] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 -> Peer 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Couldn't send request to peer 842b26ed57e14f5784632195da785fc8. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:37815: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:41.966209 31169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:41.967888 31169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:41.968384 31169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:41.971207 31169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:04:41.978168 31229 ts_tablet_manager.cc:542] Loading tablet metadata (0/5 complete)
W20260810 22:04:41.990381 31229 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet 21df603d01494efd960235764bf9e19b: could not find data dir with uuid 6c132874b6944355be3a373a1c547046
W20260810 22:04:41.991391 31231 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet 793f84134c904805bcf80ae4d941d6d4: could not find data dir with uuid 6c132874b6944355be3a373a1c547046
W20260810 22:04:41.998978 31229 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet b71556c430684bc889e3c0f013e7f38b: could not find data dir with uuid 6c132874b6944355be3a373a1c547046
I20260810 22:04:42.005550 31169 ts_tablet_manager.cc:579] Loaded tablet metadata (5 total tablets, 5 live tablets)
I20260810 22:04:42.005831 31169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.030s user 0.009s sys 0.012s
I20260810 22:04:42.006115 31169 ts_tablet_manager.cc:594] Registering tablets (0/5 complete)
I20260810 22:04:42.012569 31230 tablet_bootstrap.cc:492] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: Bootstrap starting.
E20260810 22:04:42.013116 31230 ts_tablet_manager.cc:1419] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 21df603d01494efd960235764bf9e19b
I20260810 22:04:42.013459 31230 ts_tablet_manager.cc:1397] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.002s
I20260810 22:04:42.013752 31230 tablet_replica.cc:330] stopping tablet replica
I20260810 22:04:42.014052 31230 raft_consensus.cc:2229] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:04:42.017623 31229 tablet_bootstrap.cc:492] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8: Bootstrap starting.
I20260810 22:04:42.017684 31230 raft_consensus.cc:2258] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:04:42.022588 31230 tablet_bootstrap.cc:492] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: Bootstrap starting.
E20260810 22:04:42.022987 31230 ts_tablet_manager.cc:1419] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 793f84134c904805bcf80ae4d941d6d4
I20260810 22:04:42.023240 31230 ts_tablet_manager.cc:1397] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:04:42.023475 31230 tablet_replica.cc:330] stopping tablet replica
I20260810 22:04:42.023693 31230 raft_consensus.cc:2229] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:04:42.023968 31230 raft_consensus.cc:2258] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:04:42.028688 31230 tablet_bootstrap.cc:492] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: Bootstrap starting.
E20260810 22:04:42.029063 31230 ts_tablet_manager.cc:1419] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet b71556c430684bc889e3c0f013e7f38b
I20260810 22:04:42.036948 31232 tablet_bootstrap.cc:492] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8: Bootstrap starting.
I20260810 22:04:42.029309 31230 ts_tablet_manager.cc:1397] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:04:42.045403 31169 ts_tablet_manager.cc:610] Registered 5 tablets
I20260810 22:04:42.045545 31230 tablet_replica.cc:330] stopping tablet replica
I20260810 22:04:42.045672 31169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.040s user 0.029s sys 0.001s
I20260810 22:04:42.045825 31230 raft_consensus.cc:2229] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:04:42.046239 31230 raft_consensus.cc:2258] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:04:42.101419 31229 log.cc:824] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:42.121744 31229 tablet_bootstrap.cc:492] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:04:42.123492 31229 tablet_bootstrap.cc:492] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8: Bootstrap complete.
I20260810 22:04:42.126305 31229 ts_tablet_manager.cc:1397] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8: Time spent bootstrapping tablet: real 0.109s user 0.045s sys 0.038s
I20260810 22:04:42.139680 31232 tablet_bootstrap.cc:492] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:04:42.140993 31232 tablet_bootstrap.cc:492] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8: Bootstrap complete.
I20260810 22:04:42.143208 31232 ts_tablet_manager.cc:1397] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8: Time spent bootstrapping tablet: real 0.107s user 0.029s sys 0.039s
I20260810 22:04:42.174145 31229 raft_consensus.cc:348] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:42.174163 31232 raft_consensus.cc:348] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:42.175060 31229 raft_consensus.cc:729] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842b26ed57e14f5784632195da785fc8, State: Initialized, Role: FOLLOWER
I20260810 22:04:42.175204 31232 raft_consensus.cc:729] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842b26ed57e14f5784632195da785fc8, State: Initialized, Role: FOLLOWER
I20260810 22:04:42.176101 31229 consensus_queue.cc:260] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:42.176100 31232 consensus_queue.cc:260] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:42.183894 31232 ts_tablet_manager.cc:1428] T f969babe0c5c4ae0a8a8e7162d361173 P 842b26ed57e14f5784632195da785fc8: Time spent starting tablet: real 0.040s user 0.025s sys 0.000s
I20260810 22:04:42.187652 31229 ts_tablet_manager.cc:1428] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8: Time spent starting tablet: real 0.061s user 0.024s sys 0.020s
I20260810 22:04:42.277724 31169 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.129:37815
I20260810 22:04:42.277989 31383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.129:37815 every 8 connection(s)
I20260810 22:04:42.280445 31169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
W20260810 22:04:42.284989 31385 tablet.cc:2369] T acf4ac19b8ea418aa7827d0c3df14ae0 P 842b26ed57e14f5784632195da785fc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:04:42.288120 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 31169
I20260810 22:04:42.303007 31384 heartbeater.cc:347] Connected to a master server at 127.27.17.190:33411
I20260810 22:04:42.303467 31384 heartbeater.cc:464] Registering TS with master...
I20260810 22:04:42.304529 31384 heartbeater.cc:511] Master 127.27.17.190:33411 requested a full tablet report, sending...
I20260810 22:04:42.309079 30392 ts_manager.cc:194] Re-registered known tserver with Master: 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815)
W20260810 22:04:42.311563 30392 catalog_manager.cc:5408] Tablet 21df603d01494efd960235764bf9e19b (table test-workload [id=7bdc63e81a8f42aca69adc75753bc653]) has failed on TS 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 21df603d01494efd960235764bf9e19b
W20260810 22:04:42.312265 30392 catalog_manager.cc:5408] Tablet 793f84134c904805bcf80ae4d941d6d4 (table test-workload [id=7bdc63e81a8f42aca69adc75753bc653]) has failed on TS 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 793f84134c904805bcf80ae4d941d6d4
W20260810 22:04:42.312614 30392 catalog_manager.cc:5408] Tablet b71556c430684bc889e3c0f013e7f38b (table test-workload [id=7bdc63e81a8f42aca69adc75753bc653]) has failed on TS 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet b71556c430684bc889e3c0f013e7f38b
I20260810 22:04:42.315577 30392 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.129:60443
I20260810 22:04:42.363518 30810 raft_consensus.cc:1261] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 2 FOLLOWER]: Refusing update from remote peer bd15e90d358641be9ce4e5563f57de30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260810 22:04:42.365303 31176 consensus_queue.cc:1035] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 22:04:42.389071 30914 consensus_peers.cc:487] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 -> Peer 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Couldn't send request to peer 842b26ed57e14f5784632195da785fc8. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 793f84134c904805bcf80ae4d941d6d4. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:42.389319 31176 raft_consensus.cc:998] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30: Attempting to remove follower 842b26ed57e14f5784632195da785fc8 from the Raft config. Reason: The tablet replica hosted on peer 842b26ed57e14f5784632195da785fc8 has failed
I20260810 22:04:42.391031 31176 consensus_queue.cc:237] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:42.395012 30810 raft_consensus.cc:1261] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 2 FOLLOWER]: Refusing update from remote peer bd15e90d358641be9ce4e5563f57de30: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 22:04:42.396152 31194 consensus_queue.cc:1035] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 22:04:42.401612 31176 raft_consensus.cc:2932] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } }
I20260810 22:04:42.403133 30810 raft_consensus.cc:2932] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } }
I20260810 22:04:42.413328 30392 catalog_manager.cc:5495] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 reported cstate change: config changed from index -1 to 2, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) evicted. New cstate: current_term: 2 leader_uuid: "bd15e90d358641be9ce4e5563f57de30" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } }
I20260810 22:04:42.427312 30992 consensus_queue.cc:237] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } }
W20260810 22:04:42.434064 30914 consensus_peers.cc:487] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 -> Peer 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Couldn't send request to peer 842b26ed57e14f5784632195da785fc8. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 793f84134c904805bcf80ae4d941d6d4. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:42.433175 30810 raft_consensus.cc:1261] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 2 FOLLOWER]: Refusing update from remote peer bd15e90d358641be9ce4e5563f57de30: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260810 22:04:42.436875 31194 consensus_queue.cc:1035] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 22:04:42.437852 31299 tablet_service.cc:1514] Processing DeleteTablet for tablet 793f84134c904805bcf80ae4d941d6d4 with delete_type TABLET_DATA_TOMBSTONED (TS 842b26ed57e14f5784632195da785fc8 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:35460
I20260810 22:04:42.440621 31403 ts_tablet_manager.cc:1905] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 22:04:42.443766 31194 raft_consensus.cc:2932] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } }
I20260810 22:04:42.454752 30375 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:VOTER RPC for tablet 793f84134c904805bcf80ae4d941d6d4 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260810 22:04:42.453505 30810 raft_consensus.cc:2932] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } }
I20260810 22:04:42.457643 31403 ts_tablet_manager.cc:1918] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260810 22:04:42.458024 31403 log.cc:1196] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: Deleting WAL directory at /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/wal/wals/793f84134c904805bcf80ae4d941d6d4
I20260810 22:04:42.458544 30392 catalog_manager.cc:5495] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 reported cstate change: config changed from index 2 to 3, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) added. New cstate: current_term: 2 leader_uuid: "bd15e90d358641be9ce4e5563f57de30" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } }
I20260810 22:04:42.461398 30374 catalog_manager.cc:4841] TS 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): tablet 793f84134c904805bcf80ae4d941d6d4 (table test-workload [id=7bdc63e81a8f42aca69adc75753bc653]) successfully deleted
W20260810 22:04:42.473809 30914 consensus_peers.cc:487] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 -> Peer 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Couldn't send request to peer 842b26ed57e14f5784632195da785fc8. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 21df603d01494efd960235764bf9e19b. This is attempt 6: this message will repeat every 5th retry.
I20260810 22:04:42.474215 31194 raft_consensus.cc:998] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30: Attempting to remove follower 842b26ed57e14f5784632195da785fc8 from the Raft config. Reason: The tablet replica hosted on peer 842b26ed57e14f5784632195da785fc8 has failed
I20260810 22:04:42.475780 31194 consensus_queue.cc:237] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } }
I20260810 22:04:42.480531 30810 raft_consensus.cc:1261] T 21df603d01494efd960235764bf9e19b P 219a405626974999a6bdaa700b03120e [term 1 FOLLOWER]: Refusing update from remote peer bd15e90d358641be9ce4e5563f57de30: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:04:42.481717 31176 consensus_queue.cc:1035] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 22:04:42.487720 31194 raft_consensus.cc:2932] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } }
I20260810 22:04:42.489321 30810 raft_consensus.cc:2932] T 21df603d01494efd960235764bf9e19b P 219a405626974999a6bdaa700b03120e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } }
I20260810 22:04:42.499121 30391 catalog_manager.cc:5495] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 reported cstate change: config changed from index -1 to 2, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) evicted. New cstate: current_term: 1 leader_uuid: "bd15e90d358641be9ce4e5563f57de30" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } }
I20260810 22:04:42.508916 31299 tablet_service.cc:1514] Processing DeleteTablet for tablet 21df603d01494efd960235764bf9e19b with delete_type TABLET_DATA_TOMBSTONED (TS 842b26ed57e14f5784632195da785fc8 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:35460
I20260810 22:04:42.509799 31403 ts_tablet_manager.cc:1905] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 22:04:42.512416 30992 consensus_queue.cc:237] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } }
W20260810 22:04:42.517747 30914 consensus_peers.cc:487] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 -> Peer 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Couldn't send request to peer 842b26ed57e14f5784632195da785fc8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:42.518867 30810 raft_consensus.cc:1261] T 21df603d01494efd960235764bf9e19b P 219a405626974999a6bdaa700b03120e [term 1 FOLLOWER]: Refusing update from remote peer bd15e90d358641be9ce4e5563f57de30: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 22:04:42.520068 31176 consensus_queue.cc:1035] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 22:04:42.522224 31403 ts_tablet_manager.cc:1918] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260810 22:04:42.522584 31403 log.cc:1196] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: Deleting WAL directory at /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/wal/wals/21df603d01494efd960235764bf9e19b
I20260810 22:04:42.524083 30374 catalog_manager.cc:4841] TS 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): tablet 21df603d01494efd960235764bf9e19b (table test-workload [id=7bdc63e81a8f42aca69adc75753bc653]) successfully deleted
I20260810 22:04:42.527792 31176 raft_consensus.cc:2932] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } }
I20260810 22:04:42.528723 30810 raft_consensus.cc:2932] T 21df603d01494efd960235764bf9e19b P 219a405626974999a6bdaa700b03120e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } }
I20260810 22:04:42.535615 30375 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:VOTER RPC for tablet 21df603d01494efd960235764bf9e19b with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260810 22:04:42.540311 30391 catalog_manager.cc:5495] T 21df603d01494efd960235764bf9e19b P 219a405626974999a6bdaa700b03120e reported cstate change: config changed from index 2 to 3, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) added. New cstate: current_term: 1 leader_uuid: "bd15e90d358641be9ce4e5563f57de30" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } }
I20260810 22:04:42.543851 31384 heartbeater.cc:503] Master 127.27.17.190:33411 was elected leader, sending a full tablet report...
I20260810 22:04:42.553298 31194 raft_consensus.cc:998] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30: Attempting to remove follower 842b26ed57e14f5784632195da785fc8 from the Raft config. Reason: The tablet replica hosted on peer 842b26ed57e14f5784632195da785fc8 has failed
I20260810 22:04:42.555063 31194 consensus_queue.cc:237] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:42.558957 30810 raft_consensus.cc:1261] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [term 2 FOLLOWER]: Refusing update from remote peer bd15e90d358641be9ce4e5563f57de30: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 22:04:42.560067 31194 consensus_queue.cc:1035] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260810 22:04:42.564826 30392 catalog_manager.cc:5408] Tablet 21df603d01494efd960235764bf9e19b (table test-workload [id=7bdc63e81a8f42aca69adc75753bc653]) has failed on TS 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 21df603d01494efd960235764bf9e19b
W20260810 22:04:42.565584 30392 catalog_manager.cc:5408] Tablet 793f84134c904805bcf80ae4d941d6d4 (table test-workload [id=7bdc63e81a8f42aca69adc75753bc653]) has failed on TS 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 793f84134c904805bcf80ae4d941d6d4
W20260810 22:04:42.565973 30392 catalog_manager.cc:5408] Tablet b71556c430684bc889e3c0f013e7f38b (table test-workload [id=7bdc63e81a8f42aca69adc75753bc653]) has failed on TS 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet b71556c430684bc889e3c0f013e7f38b
I20260810 22:04:42.566195 31176 raft_consensus.cc:2932] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } }
I20260810 22:04:42.574286 30810 raft_consensus.cc:2932] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } }
I20260810 22:04:42.582055 30392 catalog_manager.cc:5495] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 reported cstate change: config changed from index -1 to 2, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) evicted. New cstate: current_term: 2 leader_uuid: "bd15e90d358641be9ce4e5563f57de30" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } }
I20260810 22:04:42.595495 31299 tablet_service.cc:1514] Processing DeleteTablet for tablet b71556c430684bc889e3c0f013e7f38b with delete_type TABLET_DATA_TOMBSTONED (TS 842b26ed57e14f5784632195da785fc8 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:35460
I20260810 22:04:42.596297 31403 ts_tablet_manager.cc:1905] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 22:04:42.597640 30992 consensus_queue.cc:237] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } }
W20260810 22:04:42.603065 30914 consensus_peers.cc:487] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 -> Peer 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Couldn't send request to peer 842b26ed57e14f5784632195da785fc8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:42.604910 30810 raft_consensus.cc:1261] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [term 2 FOLLOWER]: Refusing update from remote peer bd15e90d358641be9ce4e5563f57de30: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260810 22:04:42.606577 31176 consensus_queue.cc:1035] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 22:04:42.612360 31403 ts_tablet_manager.cc:1918] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260810 22:04:42.614120 31403 log.cc:1196] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: Deleting WAL directory at /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/wal/wals/b71556c430684bc889e3c0f013e7f38b
I20260810 22:04:42.615953 31194 raft_consensus.cc:2932] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } }
I20260810 22:04:42.618227 30810 raft_consensus.cc:2932] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } }
I20260810 22:04:42.627095 30374 catalog_manager.cc:4841] TS 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): tablet b71556c430684bc889e3c0f013e7f38b (table test-workload [id=7bdc63e81a8f42aca69adc75753bc653]) successfully deleted
I20260810 22:04:42.633919 30375 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:VOTER RPC for tablet b71556c430684bc889e3c0f013e7f38b with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260810 22:04:42.635150 30392 catalog_manager.cc:5495] T b71556c430684bc889e3c0f013e7f38b P 219a405626974999a6bdaa700b03120e reported cstate change: config changed from index 2 to 3, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) added. New cstate: current_term: 2 leader_uuid: "bd15e90d358641be9ce4e5563f57de30" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } }
I20260810 22:04:42.745646 31299 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 22:04:42.746058 30781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 22:04:42.755312 30962 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20260810 22:04:42.909206 31194 raft_consensus.cc:998] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30: Attempting to remove follower 842b26ed57e14f5784632195da785fc8 from the Raft config. Reason: The tablet replica hosted on peer 842b26ed57e14f5784632195da785fc8 has failed
I20260810 22:04:42.910914 31194 consensus_queue.cc:237] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } }
I20260810 22:04:42.915896 30810 raft_consensus.cc:1261] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 2 FOLLOWER]: Refusing update from remote peer bd15e90d358641be9ce4e5563f57de30: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260810 22:04:42.917256 31176 consensus_queue.cc:1035] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 22:04:42.923487 31439 raft_consensus.cc:2932] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } }
I20260810 22:04:42.924989 30810 raft_consensus.cc:2932] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } }
I20260810 22:04:42.936043 30391 catalog_manager.cc:5495] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 reported cstate change: config changed from index 3 to 4, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) evicted. New cstate: current_term: 2 leader_uuid: "bd15e90d358641be9ce4e5563f57de30" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } }
I20260810 22:04:42.946115 31299 tablet_service.cc:1514] Processing DeleteTablet for tablet 793f84134c904805bcf80ae4d941d6d4 with delete_type TABLET_DATA_TOMBSTONED (TS 842b26ed57e14f5784632195da785fc8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35460
I20260810 22:04:42.947590 30374 catalog_manager.cc:4841] TS 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): tablet 793f84134c904805bcf80ae4d941d6d4 (table test-workload [id=7bdc63e81a8f42aca69adc75753bc653]) successfully deleted
I20260810 22:04:42.951900 30992 consensus_queue.cc:237] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } }
W20260810 22:04:42.957950 30914 consensus_peers.cc:487] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 -> Peer 842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): Couldn't send request to peer 842b26ed57e14f5784632195da785fc8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:42.958477 30810 raft_consensus.cc:1261] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 2 FOLLOWER]: Refusing update from remote peer bd15e90d358641be9ce4e5563f57de30: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260810 22:04:42.959684 31439 consensus_queue.cc:1035] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 22:04:42.965619 31176 raft_consensus.cc:2932] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } }
I20260810 22:04:42.974052 30810 raft_consensus.cc:2932] T 793f84134c904805bcf80ae4d941d6d4 P 219a405626974999a6bdaa700b03120e [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } }
I20260810 22:04:42.979012 30375 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:VOTER RPC for tablet 793f84134c904805bcf80ae4d941d6d4 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260810 22:04:42.984329 30391 catalog_manager.cc:5495] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30 reported cstate change: config changed from index 4 to 5, VOTER 842b26ed57e14f5784632195da785fc8 (127.27.17.129) added. New cstate: current_term: 2 leader_uuid: "bd15e90d358641be9ce4e5563f57de30" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } } }
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
22693aa679da4011b230e0e3fcf343fe | 127.27.17.190:33411 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:42125 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
219a405626974999a6bdaa700b03120e | 127.27.17.130:36727 | HEALTHY | <none> | 1 | 0
842b26ed57e14f5784632195da785fc8 | 127.27.17.129:37815 | HEALTHY | <none> | 0 | 0
bd15e90d358641be9ce4e5563f57de30 | 127.27.17.131:35619 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.27.17.129:37815
env_inject_eio | 1 | hidden | 127.27.17.129:37815
env_inject_eio_globs | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-1/** | hidden | 127.27.17.129:37815
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.27.17.129 | experimental | 127.27.17.129:37815
local_ip_for_outbound_sockets | 127.27.17.130 | experimental | 127.27.17.130:36727
local_ip_for_outbound_sockets | 127.27.17.131 | experimental | 127.27.17.131:35619
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.27.17.129:37815
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.27.17.130:36727
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.27.17.131:35619
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:42125 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 21df603d01494efd960235764bf9e19b of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619): RUNNING [LEADER]
219a405626974999a6bdaa700b03120e (127.27.17.130:36727): RUNNING
842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = bd15e90d358641be9ce4e5563f57de30
B = 219a405626974999a6bdaa700b03120e
C = 842b26ed57e14f5784632195da785fc8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 3 | Yes
B | A* B C | 1 | 3 | Yes
C | [config not available] | | |
Tablet 793f84134c904805bcf80ae4d941d6d4 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
219a405626974999a6bdaa700b03120e (127.27.17.130:36727): RUNNING
bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619): RUNNING [LEADER]
842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 219a405626974999a6bdaa700b03120e
B = bd15e90d358641be9ce4e5563f57de30
C = 842b26ed57e14f5784632195da785fc8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 3 | Yes
B | A B* C | 2 | 3 | Yes
C | [config not available] | | |
Tablet b71556c430684bc889e3c0f013e7f38b of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
219a405626974999a6bdaa700b03120e (127.27.17.130:36727): RUNNING
bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619): RUNNING [LEADER]
842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 219a405626974999a6bdaa700b03120e
B = bd15e90d358641be9ce4e5563f57de30
C = 842b26ed57e14f5784632195da785fc8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 3 | Yes
B | A B* C | 2 | 3 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 5 | 2 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 5
First Quartile | 5
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 22:04:43.009413 27718 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 22:04:43.091742 31441 ts_tablet_manager.cc:927] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: Initiating tablet copy from peer bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619)
I20260810 22:04:43.095293 31441 tablet_copy_client.cc:279] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid
I20260810 22:04:43.095894 31441 tablet_copy_client.cc:316] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: tablet copy: Beginning tablet copy session from remote peer at address 127.27.17.131:35619
I20260810 22:04:43.109099 31018 tablet_copy_service.cc:140] P bd15e90d358641be9ce4e5563f57de30: Received BeginTabletCopySession request for tablet 21df603d01494efd960235764bf9e19b from peer 842b26ed57e14f5784632195da785fc8 ({username='slave'} at 127.27.17.129:49667)
I20260810 22:04:43.109738 31018 tablet_copy_service.cc:161] P bd15e90d358641be9ce4e5563f57de30: Beginning new tablet copy session on tablet 21df603d01494efd960235764bf9e19b from peer 842b26ed57e14f5784632195da785fc8 at {username='slave'} at 127.27.17.129:49667: session id = 842b26ed57e14f5784632195da785fc8-21df603d01494efd960235764bf9e19b
I20260810 22:04:43.117461 31018 tablet_copy_source_session.cc:205] T 21df603d01494efd960235764bf9e19b P bd15e90d358641be9ce4e5563f57de30: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:04:43.120416 31441 ts_tablet_manager.cc:1905] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260810 22:04:43.131532 31441 ts_tablet_manager.cc:1918] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown
I20260810 22:04:43.137820 31441 tablet_copy_client.cc:791] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: tablet copy: Starting download of 0 data blocks...
I20260810 22:04:43.138407 31441 tablet_copy_client.cc:655] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: tablet copy: Starting download of 1 WAL segments...
I20260810 22:04:43.142400 31441 tablet_copy_client.cc:531] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:04:43.148164 31441 tablet_bootstrap.cc:492] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: Bootstrap starting.
I20260810 22:04:43.167845 31441 tablet_bootstrap.cc:492] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:04:43.168967 31441 tablet_bootstrap.cc:492] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: Bootstrap complete.
I20260810 22:04:43.169680 31441 ts_tablet_manager.cc:1397] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: Time spent bootstrapping tablet: real 0.022s user 0.021s sys 0.002s
I20260810 22:04:43.172014 31441 raft_consensus.cc:348] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } }
I20260810 22:04:43.172698 31441 raft_consensus.cc:729] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842b26ed57e14f5784632195da785fc8, State: Initialized, Role: FOLLOWER
I20260810 22:04:43.173305 31441 consensus_queue.cc:260] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } }
I20260810 22:04:43.175877 31441 ts_tablet_manager.cc:1428] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260810 22:04:43.177585 31018 tablet_copy_service.cc:342] P bd15e90d358641be9ce4e5563f57de30: Request end of tablet copy session 842b26ed57e14f5784632195da785fc8-21df603d01494efd960235764bf9e19b received from {username='slave'} at 127.27.17.129:49667
I20260810 22:04:43.178006 31018 tablet_copy_service.cc:433] P bd15e90d358641be9ce4e5563f57de30: ending tablet copy session 842b26ed57e14f5784632195da785fc8-21df603d01494efd960235764bf9e19b on tablet 21df603d01494efd960235764bf9e19b with peer 842b26ed57e14f5784632195da785fc8
I20260810 22:04:43.201678 31441 ts_tablet_manager.cc:927] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: Initiating tablet copy from peer bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619)
I20260810 22:04:43.203002 31441 tablet_copy_client.cc:279] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid
I20260810 22:04:43.203516 31441 tablet_copy_client.cc:316] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: tablet copy: Beginning tablet copy session from remote peer at address 127.27.17.131:35619
I20260810 22:04:43.204887 31018 tablet_copy_service.cc:140] P bd15e90d358641be9ce4e5563f57de30: Received BeginTabletCopySession request for tablet b71556c430684bc889e3c0f013e7f38b from peer 842b26ed57e14f5784632195da785fc8 ({username='slave'} at 127.27.17.129:49667)
I20260810 22:04:43.205297 31018 tablet_copy_service.cc:161] P bd15e90d358641be9ce4e5563f57de30: Beginning new tablet copy session on tablet b71556c430684bc889e3c0f013e7f38b from peer 842b26ed57e14f5784632195da785fc8 at {username='slave'} at 127.27.17.129:49667: session id = 842b26ed57e14f5784632195da785fc8-b71556c430684bc889e3c0f013e7f38b
I20260810 22:04:43.209913 31018 tablet_copy_source_session.cc:205] T b71556c430684bc889e3c0f013e7f38b P bd15e90d358641be9ce4e5563f57de30: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:04:43.211858 31441 ts_tablet_manager.cc:1905] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260810 22:04:43.221777 31441 ts_tablet_manager.cc:1918] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown
I20260810 22:04:43.228230 31441 tablet_copy_client.cc:791] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: tablet copy: Starting download of 0 data blocks...
I20260810 22:04:43.228778 31441 tablet_copy_client.cc:655] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: tablet copy: Starting download of 1 WAL segments...
I20260810 22:04:43.232532 31441 tablet_copy_client.cc:531] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:04:43.238782 31441 tablet_bootstrap.cc:492] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: Bootstrap starting.
I20260810 22:04:43.259171 31441 tablet_bootstrap.cc:492] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:04:43.260224 31441 tablet_bootstrap.cc:492] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: Bootstrap complete.
I20260810 22:04:43.260802 31441 ts_tablet_manager.cc:1397] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.008s
I20260810 22:04:43.262508 31441 raft_consensus.cc:348] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } }
I20260810 22:04:43.262943 31441 raft_consensus.cc:729] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842b26ed57e14f5784632195da785fc8, State: Initialized, Role: FOLLOWER
I20260810 22:04:43.263485 31441 consensus_queue.cc:260] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } }
I20260810 22:04:43.264995 31441 ts_tablet_manager.cc:1428] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260810 22:04:43.266566 31018 tablet_copy_service.cc:342] P bd15e90d358641be9ce4e5563f57de30: Request end of tablet copy session 842b26ed57e14f5784632195da785fc8-b71556c430684bc889e3c0f013e7f38b received from {username='slave'} at 127.27.17.129:49667
I20260810 22:04:43.266958 31018 tablet_copy_service.cc:433] P bd15e90d358641be9ce4e5563f57de30: ending tablet copy session 842b26ed57e14f5784632195da785fc8-b71556c430684bc889e3c0f013e7f38b on tablet b71556c430684bc889e3c0f013e7f38b with peer 842b26ed57e14f5784632195da785fc8
E20260810 22:04:43.311929 31378 env_posix.cc:432] I/O error, context: INJECTED FAILURE
I20260810 22:04:43.417109 30781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 22:04:43.421977 31299 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 22:04:43.423195 30962 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20260810 22:04:43.518002 31441 ts_tablet_manager.cc:927] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: Initiating tablet copy from peer bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619)
I20260810 22:04:43.529932 31441 tablet_copy_client.cc:279] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid
I20260810 22:04:43.531978 31441 tablet_copy_client.cc:316] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: tablet copy: Beginning tablet copy session from remote peer at address 127.27.17.131:35619
I20260810 22:04:43.533630 31018 tablet_copy_service.cc:140] P bd15e90d358641be9ce4e5563f57de30: Received BeginTabletCopySession request for tablet 793f84134c904805bcf80ae4d941d6d4 from peer 842b26ed57e14f5784632195da785fc8 ({username='slave'} at 127.27.17.129:49667)
I20260810 22:04:43.534099 31018 tablet_copy_service.cc:161] P bd15e90d358641be9ce4e5563f57de30: Beginning new tablet copy session on tablet 793f84134c904805bcf80ae4d941d6d4 from peer 842b26ed57e14f5784632195da785fc8 at {username='slave'} at 127.27.17.129:49667: session id = 842b26ed57e14f5784632195da785fc8-793f84134c904805bcf80ae4d941d6d4
I20260810 22:04:43.539135 31018 tablet_copy_source_session.cc:205] T 793f84134c904805bcf80ae4d941d6d4 P bd15e90d358641be9ce4e5563f57de30: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:04:43.541610 31441 ts_tablet_manager.cc:1905] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260810 22:04:43.554965 31441 ts_tablet_manager.cc:1918] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown
I20260810 22:04:43.562649 31441 tablet_copy_client.cc:791] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: tablet copy: Starting download of 0 data blocks...
I20260810 22:04:43.563262 31441 tablet_copy_client.cc:655] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: tablet copy: Starting download of 1 WAL segments...
I20260810 22:04:43.569478 31441 tablet_copy_client.cc:531] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:04:43.577041 31441 tablet_bootstrap.cc:492] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: Bootstrap starting.
I20260810 22:04:43.602388 31441 tablet_bootstrap.cc:492] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:04:43.603371 31441 tablet_bootstrap.cc:492] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: Bootstrap complete.
I20260810 22:04:43.604040 31441 ts_tablet_manager.cc:1397] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: Time spent bootstrapping tablet: real 0.027s user 0.029s sys 0.000s
I20260810 22:04:43.606263 31441 raft_consensus.cc:348] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } }
I20260810 22:04:43.606879 31441 raft_consensus.cc:729] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842b26ed57e14f5784632195da785fc8, State: Initialized, Role: FOLLOWER
I20260810 22:04:43.607457 31441 consensus_queue.cc:260] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "219a405626974999a6bdaa700b03120e" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 36727 } } peers { permanent_uuid: "bd15e90d358641be9ce4e5563f57de30" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 35619 } } peers { permanent_uuid: "842b26ed57e14f5784632195da785fc8" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 37815 } }
I20260810 22:04:43.609166 31441 ts_tablet_manager.cc:1428] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260810 22:04:43.611161 31018 tablet_copy_service.cc:342] P bd15e90d358641be9ce4e5563f57de30: Request end of tablet copy session 842b26ed57e14f5784632195da785fc8-793f84134c904805bcf80ae4d941d6d4 received from {username='slave'} at 127.27.17.129:49667
I20260810 22:04:43.611506 31018 tablet_copy_service.cc:433] P bd15e90d358641be9ce4e5563f57de30: ending tablet copy session 842b26ed57e14f5784632195da785fc8-793f84134c904805bcf80ae4d941d6d4 on tablet 793f84134c904805bcf80ae4d941d6d4 with peer 842b26ed57e14f5784632195da785fc8
I20260810 22:04:43.615505 31327 raft_consensus.cc:1203] T 21df603d01494efd960235764bf9e19b P 842b26ed57e14f5784632195da785fc8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
22693aa679da4011b230e0e3fcf343fe | 127.27.17.190:33411 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:42125 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
219a405626974999a6bdaa700b03120e | 127.27.17.130:36727 | HEALTHY | <none> | 1 | 0
842b26ed57e14f5784632195da785fc8 | 127.27.17.129:37815 | HEALTHY | <none> | 0 | 0
bd15e90d358641be9ce4e5563f57de30 | 127.27.17.131:35619 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.27.17.129:37815
env_inject_eio | 1 | hidden | 127.27.17.129:37815
env_inject_eio_globs | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-1/** | hidden | 127.27.17.129:37815
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.27.17.129 | experimental | 127.27.17.129:37815
local_ip_for_outbound_sockets | 127.27.17.130 | experimental | 127.27.17.130:36727
local_ip_for_outbound_sockets | 127.27.17.131 | experimental | 127.27.17.131:35619
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.27.17.129:37815
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.27.17.130:36727
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.27.17.131:35619
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:42125 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 21df603d01494efd960235764bf9e19b of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619): RUNNING [LEADER]
219a405626974999a6bdaa700b03120e (127.27.17.130:36727): RUNNING
842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): RUNNING
All reported replicas are:
A = bd15e90d358641be9ce4e5563f57de30
B = 219a405626974999a6bdaa700b03120e
C = 842b26ed57e14f5784632195da785fc8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 3 | Yes
B | A* B C | 1 | 3 | Yes
C | A B C | 1 | 3 | Yes
Tablet 793f84134c904805bcf80ae4d941d6d4 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
219a405626974999a6bdaa700b03120e (127.27.17.130:36727): RUNNING
bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619): RUNNING [LEADER]
842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 219a405626974999a6bdaa700b03120e
B = bd15e90d358641be9ce4e5563f57de30
C = 842b26ed57e14f5784632195da785fc8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 5 | Yes
B | A B* C | 2 | 5 | Yes
C | [config not available] | | |
Tablet b71556c430684bc889e3c0f013e7f38b of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
219a405626974999a6bdaa700b03120e (127.27.17.130:36727): RUNNING
bd15e90d358641be9ce4e5563f57de30 (127.27.17.131:35619): RUNNING [LEADER]
842b26ed57e14f5784632195da785fc8 (127.27.17.129:37815): RUNNING
All reported replicas are:
A = 219a405626974999a6bdaa700b03120e
B = bd15e90d358641be9ce4e5563f57de30
C = 842b26ed57e14f5784632195da785fc8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 3 | Yes
B | A B* C | 2 | 3 | Yes
C | A B C | 2 | 3 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 5 | 2 | 0 | 1 | 2
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 5
First Quartile | 5
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 22:04:43.653563 27718 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 22:04:43.782274 31327 raft_consensus.cc:1203] T b71556c430684bc889e3c0f013e7f38b P 842b26ed57e14f5784632195da785fc8 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260810 22:04:43.928268 31327 raft_consensus.cc:1203] T 793f84134c904805bcf80ae4d941d6d4 P 842b26ed57e14f5784632195da785fc8 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.4->[2.5-2.5] Dedup: 2.5->[]
I20260810 22:04:44.171413 30781 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 22:04:44.184628 30962 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20260810 22:04:44.190667 31299 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
22693aa679da4011b230e0e3fcf343fe | 127.27.17.190:33411 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:42125 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
219a405626974999a6bdaa700b03120e | 127.27.17.130:36727 | HEALTHY | <none> | 1 | 0
842b26ed57e14f5784632195da785fc8 | 127.27.17.129:37815 | HEALTHY | <none> | 0 | 0
bd15e90d358641be9ce4e5563f57de30 | 127.27.17.131:35619 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.27.17.129:37815
env_inject_eio | 1 | hidden | 127.27.17.129:37815
env_inject_eio_globs | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-1/** | hidden | 127.27.17.129:37815
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.27.17.129 | experimental | 127.27.17.129:37815
local_ip_for_outbound_sockets | 127.27.17.130 | experimental | 127.27.17.130:36727
local_ip_for_outbound_sockets | 127.27.17.131 | experimental | 127.27.17.131:35619
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.27.17.129:37815
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.27.17.130:36727
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.27.17.131:35619
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:42125 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 5 | 5 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 5
First Quartile | 5
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260810 22:04:44.436585 27718 log_verifier.cc:126] Checking tablet 21df603d01494efd960235764bf9e19b
I20260810 22:04:44.499387 27718 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 21df603d01494efd960235764bf9e19b
I20260810 22:04:44.499670 27718 log_verifier.cc:126] Checking tablet 793f84134c904805bcf80ae4d941d6d4
I20260810 22:04:44.563656 27718 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 793f84134c904805bcf80ae4d941d6d4
I20260810 22:04:44.563932 27718 log_verifier.cc:126] Checking tablet acf4ac19b8ea418aa7827d0c3df14ae0
I20260810 22:04:44.622107 27718 log_verifier.cc:177] Verified matching terms for 1 ops in tablet acf4ac19b8ea418aa7827d0c3df14ae0
I20260810 22:04:44.622344 27718 log_verifier.cc:126] Checking tablet b71556c430684bc889e3c0f013e7f38b
I20260810 22:04:44.683178 27718 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b71556c430684bc889e3c0f013e7f38b
I20260810 22:04:44.683447 27718 log_verifier.cc:126] Checking tablet f969babe0c5c4ae0a8a8e7162d361173
I20260810 22:04:44.741863 27718 log_verifier.cc:177] Verified matching terms for 1 ops in tablet f969babe0c5c4ae0a8a8e7162d361173
I20260810 22:04:45.195979 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 31169
I20260810 22:04:45.252328 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 30687
W20260810 22:04:45.291508 30914 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.129:37815: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20260810 22:04:45.295297 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 30877
I20260810 22:04:45.341722 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 30335
2026-08-10T22:04:45Z chronyd exiting
[ OK ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/2 (14214 ms)
[ RUN ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/3
2026-08-10T22:04:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:04:45Z Disabled control of system clock
I20260810 22:04:45.470773 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.190:44707
--webserver_interface=127.27.17.190
--webserver_port=0
--builtin_ntp_servers=127.27.17.148:38205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.190:44707
--raft_prepare_replacement_before_eviction=true with env {}
W20260810 22:04:45.802378 31554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:45.803049 31554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:45.803550 31554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:45.835366 31554 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20260810 22:04:45.835784 31554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:04:45.836004 31554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:45.836200 31554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:04:45.836388 31554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:04:45.871513 31554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:38205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.190:44707
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.190:44707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
--webserver_interface=127.27.17.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:45.872730 31554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:04:45.874403 31554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:45.893582 31566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:45.893589 31562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:45.894048 31563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:47.098866 31565 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1206 milliseconds
I20260810 22:04:47.098990 31554 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 22:04:47.100361 31554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:47.103621 31554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:47.105100 31554 hybrid_clock.cc:648] HybridClock initialized: now 1786399487105047 us; error 67 us; skew 500 ppm
I20260810 22:04:47.106051 31554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:47.113324 31554 webserver.cc:460] Webserver started at http://127.27.17.190:38741/ using document root <none> and password file <none>
I20260810 22:04:47.114979 31554 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:47.115217 31554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:04:47.115840 31554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:04:47.120959 31554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-0/instance:
uuid: "14ffa66e458544e5900503af3474f918"
format_stamp: "Formatted at 2026-08-10 22:04:47 on dist-test-slave-bw88"
I20260810 22:04:47.122189 31554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-1/instance:
uuid: "14ffa66e458544e5900503af3474f918"
format_stamp: "Formatted at 2026-08-10 22:04:47 on dist-test-slave-bw88"
I20260810 22:04:47.123055 31554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-2/instance:
uuid: "14ffa66e458544e5900503af3474f918"
format_stamp: "Formatted at 2026-08-10 22:04:47 on dist-test-slave-bw88"
I20260810 22:04:47.123890 31554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-3/instance:
uuid: "14ffa66e458544e5900503af3474f918"
format_stamp: "Formatted at 2026-08-10 22:04:47 on dist-test-slave-bw88"
I20260810 22:04:47.124799 31554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-4/instance:
uuid: "14ffa66e458544e5900503af3474f918"
format_stamp: "Formatted at 2026-08-10 22:04:47 on dist-test-slave-bw88"
I20260810 22:04:47.125682 31554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/wal/instance:
uuid: "14ffa66e458544e5900503af3474f918"
format_stamp: "Formatted at 2026-08-10 22:04:47 on dist-test-slave-bw88"
I20260810 22:04:47.149495 31554 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.012s sys 0.012s
I20260810 22:04:47.157459 31578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:47.158181 31578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:47.158650 31578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:47.158866 31579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:47.159050 31578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:47.161681 31554 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.004s
I20260810 22:04:47.162000 31554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/wal
uuid: "14ffa66e458544e5900503af3474f918"
format_stamp: "Formatted at 2026-08-10 22:04:47 on dist-test-slave-bw88"
I20260810 22:04:47.162385 31554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:04:47.206161 31554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:47.207633 31554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:47.208071 31554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:47.279331 31554 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.190:44707
I20260810 22:04:47.279794 31655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.190:44707 every 8 connection(s)
I20260810 22:04:47.282341 31554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
I20260810 22:04:47.291865 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 31554
I20260810 22:04:47.292397 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/wal/instance
I20260810 22:04:47.310962 31656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918: Bootstrap starting.
I20260810 22:04:47.317674 31656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:47.319717 31656 log.cc:824] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:47.324558 31656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918: No bootstrap required, opened a new log
I20260810 22:04:47.344537 31656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14ffa66e458544e5900503af3474f918" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 44707 } }
I20260810 22:04:47.345232 31656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:47.345542 31656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ffa66e458544e5900503af3474f918, State: Initialized, Role: FOLLOWER
I20260810 22:04:47.346359 31656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14ffa66e458544e5900503af3474f918" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 44707 } }
I20260810 22:04:47.346912 31656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:04:47.347201 31656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:04:47.347517 31656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:47.351725 31656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14ffa66e458544e5900503af3474f918" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 44707 } }
I20260810 22:04:47.352417 31656 leader_election.cc:302] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14ffa66e458544e5900503af3474f918; no voters:
I20260810 22:04:47.354157 31656 leader_election.cc:288] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:04:47.354566 31669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:47.357144 31669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918 [term 1 LEADER]: Becoming Leader. State: Replica: 14ffa66e458544e5900503af3474f918, State: Running, Role: LEADER
I20260810 22:04:47.357941 31669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14ffa66e458544e5900503af3474f918" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 44707 } }
I20260810 22:04:47.358872 31656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:04:47.369442 31673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14ffa66e458544e5900503af3474f918. Latest consensus state: current_term: 1 leader_uuid: "14ffa66e458544e5900503af3474f918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14ffa66e458544e5900503af3474f918" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 44707 } } }
I20260810 22:04:47.370421 31673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918 [sys.catalog]: This master's current role is: LEADER
I20260810 22:04:47.373994 31672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14ffa66e458544e5900503af3474f918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14ffa66e458544e5900503af3474f918" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 44707 } } }
I20260810 22:04:47.374864 31672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918 [sys.catalog]: This master's current role is: LEADER
I20260810 22:04:47.381852 31683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:04:47.391041 31683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:04:47.407353 31683 catalog_manager.cc:1347] Generated new cluster ID: de6988ae1bf2453480b6056cf319b971
I20260810 22:04:47.407697 31683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:04:47.433048 31683 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:04:47.434998 31683 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:04:47.449954 31683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14ffa66e458544e5900503af3474f918: Generated new TSK 0
I20260810 22:04:47.451017 31683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:04:47.471614 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.129:0
--local_ip_for_outbound_sockets=127.27.17.129
--webserver_interface=127.27.17.129
--webserver_port=0
--tserver_master_addrs=127.27.17.190:44707
--builtin_ntp_servers=127.27.17.148:38205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20260810 22:04:47.783990 31694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:47.784547 31694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:47.785048 31694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:47.815306 31694 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20260810 22:04:47.815735 31694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:47.816939 31694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.129
I20260810 22:04:47.851677 31694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:38205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.27.17.129
--webserver_port=0
--tserver_master_addrs=127.27.17.190:44707
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.129
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:47.852981 31694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:04:47.854605 31694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:47.875526 31703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:47.877553 31694 server_base.cc:1034] running on GCE node
W20260810 22:04:47.877638 31702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:47.875669 31706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:49.077008 31694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:49.079989 31694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:49.081522 31694 hybrid_clock.cc:648] HybridClock initialized: now 1786399489081460 us; error 48 us; skew 500 ppm
I20260810 22:04:49.082398 31694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:49.101148 31694 webserver.cc:460] Webserver started at http://127.27.17.129:38519/ using document root <none> and password file <none>
I20260810 22:04:49.102840 31694 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:49.103093 31694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:04:49.103827 31694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:04:49.109771 31694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-0/instance:
uuid: "dee255d32da241aeb929742ee6005b22"
format_stamp: "Formatted at 2026-08-10 22:04:49 on dist-test-slave-bw88"
I20260810 22:04:49.110978 31694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-1/instance:
uuid: "dee255d32da241aeb929742ee6005b22"
format_stamp: "Formatted at 2026-08-10 22:04:49 on dist-test-slave-bw88"
I20260810 22:04:49.112133 31694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-2/instance:
uuid: "dee255d32da241aeb929742ee6005b22"
format_stamp: "Formatted at 2026-08-10 22:04:49 on dist-test-slave-bw88"
I20260810 22:04:49.113123 31694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-3/instance:
uuid: "dee255d32da241aeb929742ee6005b22"
format_stamp: "Formatted at 2026-08-10 22:04:49 on dist-test-slave-bw88"
I20260810 22:04:49.114125 31694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-4/instance:
uuid: "dee255d32da241aeb929742ee6005b22"
format_stamp: "Formatted at 2026-08-10 22:04:49 on dist-test-slave-bw88"
I20260810 22:04:49.115093 31694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/wal/instance:
uuid: "dee255d32da241aeb929742ee6005b22"
format_stamp: "Formatted at 2026-08-10 22:04:49 on dist-test-slave-bw88"
I20260810 22:04:49.141268 31694 fs_manager.cc:691] Time spent creating directory manager: real 0.025s user 0.022s sys 0.005s
I20260810 22:04:49.150315 31724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:49.150840 31724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:49.152284 31724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:49.152650 31724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:49.152765 31726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:49.155256 31694 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.006s sys 0.002s
I20260810 22:04:49.155642 31694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/wal
uuid: "dee255d32da241aeb929742ee6005b22"
format_stamp: "Formatted at 2026-08-10 22:04:49 on dist-test-slave-bw88"
I20260810 22:04:49.156085 31694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:04:49.214891 31694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:49.216580 31694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:49.217108 31694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:49.220064 31694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:04:49.225216 31694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:04:49.225490 31694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 22:04:49.225831 31694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:04:49.225997 31694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:49.374032 31694 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.129:34103
I20260810 22:04:49.374578 31888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.129:34103 every 8 connection(s)
I20260810 22:04:49.376785 31694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
I20260810 22:04:49.383250 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 31694
I20260810 22:04:49.383811 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/wal/instance
I20260810 22:04:49.390612 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.130:0
--local_ip_for_outbound_sockets=127.27.17.130
--webserver_interface=127.27.17.130
--webserver_port=0
--tserver_master_addrs=127.27.17.190:44707
--builtin_ntp_servers=127.27.17.148:38205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260810 22:04:49.403805 31889 heartbeater.cc:347] Connected to a master server at 127.27.17.190:44707
I20260810 22:04:49.404362 31889 heartbeater.cc:464] Registering TS with master...
I20260810 22:04:49.405717 31889 heartbeater.cc:511] Master 127.27.17.190:44707 requested a full tablet report, sending...
I20260810 22:04:49.408810 31607 ts_manager.cc:194] Registered new tserver with Master: dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103)
I20260810 22:04:49.411132 31607 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.129:46153
W20260810 22:04:49.708570 31893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:49.709118 31893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:49.709704 31893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:49.740212 31893 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20260810 22:04:49.740687 31893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:49.741544 31893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.130
I20260810 22:04:49.775832 31893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:38205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
--webserver_interface=127.27.17.130
--webserver_port=0
--tserver_master_addrs=127.27.17.190:44707
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.130
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:49.777135 31893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:04:49.778795 31893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:49.799281 31902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:50.417505 31889 heartbeater.cc:503] Master 127.27.17.190:44707 was elected leader, sending a full tablet report...
W20260810 22:04:49.799476 31905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:49.800652 31907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:51.021950 31906 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 22:04:51.021998 31893 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 22:04:51.025856 31893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:51.028743 31893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:51.030194 31893 hybrid_clock.cc:648] HybridClock initialized: now 1786399491030133 us; error 79 us; skew 500 ppm
I20260810 22:04:51.030969 31893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:51.038206 31893 webserver.cc:460] Webserver started at http://127.27.17.130:42355/ using document root <none> and password file <none>
I20260810 22:04:51.039624 31893 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:51.039816 31893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:04:51.040403 31893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:04:51.045259 31893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-0/instance:
uuid: "44c1b033bb9541a7bc66207ff2deba61"
format_stamp: "Formatted at 2026-08-10 22:04:51 on dist-test-slave-bw88"
I20260810 22:04:51.046727 31893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-1/instance:
uuid: "44c1b033bb9541a7bc66207ff2deba61"
format_stamp: "Formatted at 2026-08-10 22:04:51 on dist-test-slave-bw88"
I20260810 22:04:51.047567 31893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-2/instance:
uuid: "44c1b033bb9541a7bc66207ff2deba61"
format_stamp: "Formatted at 2026-08-10 22:04:51 on dist-test-slave-bw88"
I20260810 22:04:51.048411 31893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-3/instance:
uuid: "44c1b033bb9541a7bc66207ff2deba61"
format_stamp: "Formatted at 2026-08-10 22:04:51 on dist-test-slave-bw88"
I20260810 22:04:51.049230 31893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-4/instance:
uuid: "44c1b033bb9541a7bc66207ff2deba61"
format_stamp: "Formatted at 2026-08-10 22:04:51 on dist-test-slave-bw88"
I20260810 22:04:51.050119 31893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/wal/instance:
uuid: "44c1b033bb9541a7bc66207ff2deba61"
format_stamp: "Formatted at 2026-08-10 22:04:51 on dist-test-slave-bw88"
I20260810 22:04:51.074143 31893 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.015s sys 0.009s
I20260810 22:04:51.082460 31923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:51.083022 31923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:51.084301 31923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:51.084540 31924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:51.084754 31923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:51.087716 31893 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.005s sys 0.004s
I20260810 22:04:51.088229 31893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/wal
uuid: "44c1b033bb9541a7bc66207ff2deba61"
format_stamp: "Formatted at 2026-08-10 22:04:51 on dist-test-slave-bw88"
I20260810 22:04:51.088856 31893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:04:51.121053 31893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:51.122606 31893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:51.123059 31893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:51.125754 31893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:04:51.130338 31893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:04:51.130589 31893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:51.130841 31893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:04:51.131016 31893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:51.276177 31893 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.130:46865
I20260810 22:04:51.276674 32069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.130:46865 every 8 connection(s)
I20260810 22:04:51.279860 31893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
I20260810 22:04:51.281805 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 31893
I20260810 22:04:51.282750 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/wal/instance
I20260810 22:04:51.294683 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.131:0
--local_ip_for_outbound_sockets=127.27.17.131
--webserver_interface=127.27.17.131
--webserver_port=0
--tserver_master_addrs=127.27.17.190:44707
--builtin_ntp_servers=127.27.17.148:38205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260810 22:04:51.333377 32072 heartbeater.cc:347] Connected to a master server at 127.27.17.190:44707
I20260810 22:04:51.333829 32072 heartbeater.cc:464] Registering TS with master...
I20260810 22:04:51.334822 32072 heartbeater.cc:511] Master 127.27.17.190:44707 requested a full tablet report, sending...
I20260810 22:04:51.337121 31607 ts_manager.cc:194] Registered new tserver with Master: 44c1b033bb9541a7bc66207ff2deba61 (127.27.17.130:46865)
I20260810 22:04:51.339128 31607 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.130:52341
W20260810 22:04:51.626371 32077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:51.626881 32077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:51.627317 32077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:51.659198 32077 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20260810 22:04:51.659579 32077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:51.660429 32077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.131
I20260810 22:04:51.696233 32077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:38205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
--webserver_interface=127.27.17.131
--webserver_port=0
--tserver_master_addrs=127.27.17.190:44707
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.131
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:51.697476 32077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:04:51.699115 32077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:51.718776 32088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:52.343053 32072 heartbeater.cc:503] Master 127.27.17.190:44707 was elected leader, sending a full tablet report...
W20260810 22:04:51.718851 32090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:51.720026 32087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:52.904623 32089 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 22:04:52.904685 32077 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 22:04:52.908875 32077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:52.912101 32077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:52.913558 32077 hybrid_clock.cc:648] HybridClock initialized: now 1786399492913495 us; error 85 us; skew 500 ppm
I20260810 22:04:52.914417 32077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:52.921233 32077 webserver.cc:460] Webserver started at http://127.27.17.131:38829/ using document root <none> and password file <none>
I20260810 22:04:52.922784 32077 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:52.923022 32077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:04:52.923697 32077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:04:52.929687 32077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-0/instance:
uuid: "e07d7c5a69cf4e2e938b11c692e3565c"
format_stamp: "Formatted at 2026-08-10 22:04:52 on dist-test-slave-bw88"
I20260810 22:04:52.930860 32077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-1/instance:
uuid: "e07d7c5a69cf4e2e938b11c692e3565c"
format_stamp: "Formatted at 2026-08-10 22:04:52 on dist-test-slave-bw88"
I20260810 22:04:52.931772 32077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-2/instance:
uuid: "e07d7c5a69cf4e2e938b11c692e3565c"
format_stamp: "Formatted at 2026-08-10 22:04:52 on dist-test-slave-bw88"
I20260810 22:04:52.932662 32077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-3/instance:
uuid: "e07d7c5a69cf4e2e938b11c692e3565c"
format_stamp: "Formatted at 2026-08-10 22:04:52 on dist-test-slave-bw88"
I20260810 22:04:52.933575 32077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-4/instance:
uuid: "e07d7c5a69cf4e2e938b11c692e3565c"
format_stamp: "Formatted at 2026-08-10 22:04:52 on dist-test-slave-bw88"
I20260810 22:04:52.934453 32077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/wal/instance:
uuid: "e07d7c5a69cf4e2e938b11c692e3565c"
format_stamp: "Formatted at 2026-08-10 22:04:52 on dist-test-slave-bw88"
I20260810 22:04:52.958206 32077 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.015s sys 0.008s
I20260810 22:04:52.966719 32104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:52.967278 32104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:52.968427 32106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:52.968544 32104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:52.968904 32106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:52.971148 32077 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.006s
I20260810 22:04:52.971490 32077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/wal
uuid: "e07d7c5a69cf4e2e938b11c692e3565c"
format_stamp: "Formatted at 2026-08-10 22:04:52 on dist-test-slave-bw88"
I20260810 22:04:52.971907 32077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:04:53.018157 32077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:53.019652 32077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:53.020078 32077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:53.022856 32077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:04:53.027518 32077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:04:53.027769 32077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:53.028026 32077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:04:53.028203 32077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:53.185034 32077 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.131:38509
I20260810 22:04:53.185565 32255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.131:38509 every 8 connection(s)
I20260810 22:04:53.187816 32077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
I20260810 22:04:53.194800 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 32077
I20260810 22:04:53.195259 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/wal/instance
I20260810 22:04:53.201660 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.132:0
--local_ip_for_outbound_sockets=127.27.17.132
--webserver_interface=127.27.17.132
--webserver_port=0
--tserver_master_addrs=127.27.17.190:44707
--builtin_ntp_servers=127.27.17.148:38205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260810 22:04:53.216110 32256 heartbeater.cc:347] Connected to a master server at 127.27.17.190:44707
I20260810 22:04:53.216675 32256 heartbeater.cc:464] Registering TS with master...
I20260810 22:04:53.218101 32256 heartbeater.cc:511] Master 127.27.17.190:44707 requested a full tablet report, sending...
I20260810 22:04:53.221069 31607 ts_manager.cc:194] Registered new tserver with Master: e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131:38509)
I20260810 22:04:53.222597 31607 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.131:45711
W20260810 22:04:53.525918 32260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:53.526459 32260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:53.526990 32260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:53.557669 32260 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20260810 22:04:53.558072 32260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:53.558820 32260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.132
I20260810 22:04:53.594040 32260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:38205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-0/info.pb
--webserver_interface=127.27.17.132
--webserver_port=0
--tserver_master_addrs=127.27.17.190:44707
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.132
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:53.595255 32260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:04:53.596786 32260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:53.616323 32269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:54.227614 32256 heartbeater.cc:503] Master 127.27.17.190:44707 was elected leader, sending a full tablet report...
W20260810 22:04:53.616349 32271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:53.617964 32260 server_base.cc:1034] running on GCE node
W20260810 22:04:53.616394 32268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:54.837998 32260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:54.840262 32260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:54.841642 32260 hybrid_clock.cc:648] HybridClock initialized: now 1786399494841592 us; error 45 us; skew 500 ppm
I20260810 22:04:54.842424 32260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:54.848663 32260 webserver.cc:460] Webserver started at http://127.27.17.132:34275/ using document root <none> and password file <none>
I20260810 22:04:54.850152 32260 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:54.850343 32260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:04:54.851035 32260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:04:54.855957 32260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-0/instance:
uuid: "dd8ff3433b064de08a86dce7ab10aab9"
format_stamp: "Formatted at 2026-08-10 22:04:54 on dist-test-slave-bw88"
I20260810 22:04:54.857067 32260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-1/instance:
uuid: "dd8ff3433b064de08a86dce7ab10aab9"
format_stamp: "Formatted at 2026-08-10 22:04:54 on dist-test-slave-bw88"
I20260810 22:04:54.857976 32260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-2/instance:
uuid: "dd8ff3433b064de08a86dce7ab10aab9"
format_stamp: "Formatted at 2026-08-10 22:04:54 on dist-test-slave-bw88"
I20260810 22:04:54.858788 32260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-3/instance:
uuid: "dd8ff3433b064de08a86dce7ab10aab9"
format_stamp: "Formatted at 2026-08-10 22:04:54 on dist-test-slave-bw88"
I20260810 22:04:54.859745 32260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-4/instance:
uuid: "dd8ff3433b064de08a86dce7ab10aab9"
format_stamp: "Formatted at 2026-08-10 22:04:54 on dist-test-slave-bw88"
I20260810 22:04:54.860661 32260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/wal/instance:
uuid: "dd8ff3433b064de08a86dce7ab10aab9"
format_stamp: "Formatted at 2026-08-10 22:04:54 on dist-test-slave-bw88"
I20260810 22:04:54.884451 32260 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.021s sys 0.002s
I20260810 22:04:54.892573 32288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:54.893213 32288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:54.894414 32288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:54.894814 32288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:54.894812 32289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:54.897150 32260 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.006s sys 0.004s
I20260810 22:04:54.897588 32260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/wal
uuid: "dd8ff3433b064de08a86dce7ab10aab9"
format_stamp: "Formatted at 2026-08-10 22:04:54 on dist-test-slave-bw88"
I20260810 22:04:54.898146 32260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:04:54.934314 32260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:54.936434 32260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:54.936908 32260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:54.939615 32260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:04:54.944336 32260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:04:54.944589 32260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:54.944840 32260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:04:54.945015 32260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:55.103132 32260 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.132:34125
I20260810 22:04:55.103624 32450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.132:34125 every 8 connection(s)
I20260810 22:04:55.106555 32260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-0/info.pb
I20260810 22:04:55.108217 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 32260
I20260810 22:04:55.108911 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/wal/instance
I20260810 22:04:55.132607 32453 heartbeater.cc:347] Connected to a master server at 127.27.17.190:44707
I20260810 22:04:55.133072 32453 heartbeater.cc:464] Registering TS with master...
I20260810 22:04:55.134087 32453 heartbeater.cc:511] Master 127.27.17.190:44707 requested a full tablet report, sending...
I20260810 22:04:55.136209 31607 ts_manager.cc:194] Registered new tserver with Master: dd8ff3433b064de08a86dce7ab10aab9 (127.27.17.132:34125)
I20260810 22:04:55.137925 31607 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.132:46179
I20260810 22:04:55.146021 27718 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20260810 22:04:55.182559 27718 test_util.cc:268] Using random seed: -1957275451
I20260810 22:04:55.234099 31606 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48072:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\231\231\231\031\004\001\0002333\004\001\000\313\314\314L\004\001\000dfff"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 22:04:55.336030 32356 tablet_service.cc:1467] Processing CreateTablet for tablet f4bc97c76ce24181bbac8c1b6c0da80e (DEFAULT_TABLE table=test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260810 22:04:55.337672 32355 tablet_service.cc:1467] Processing CreateTablet for tablet 26ac18eeb4a54ec2865d5bed0b18c84a (DEFAULT_TABLE table=test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260810 22:04:55.335520 32359 tablet_service.cc:1467] Processing CreateTablet for tablet 0bac2f72b88b4c74a8ef33e484828638 (DEFAULT_TABLE table=test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260810 22:04:55.353402 31988 tablet_service.cc:1467] Processing CreateTablet for tablet 0bac2f72b88b4c74a8ef33e484828638 (DEFAULT_TABLE table=test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260810 22:04:55.354193 31987 tablet_service.cc:1467] Processing CreateTablet for tablet f5b7bfea936e4001887b340805850f7a (DEFAULT_TABLE table=test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260810 22:04:55.352479 31985 tablet_service.cc:1467] Processing CreateTablet for tablet 79023ddd04404bc0ba3d3e673cd80c08 (DEFAULT_TABLE table=test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260810 22:04:55.354384 32171 tablet_service.cc:1467] Processing CreateTablet for tablet 0bac2f72b88b4c74a8ef33e484828638 (DEFAULT_TABLE table=test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260810 22:04:55.356892 31986 tablet_service.cc:1467] Processing CreateTablet for tablet f4bc97c76ce24181bbac8c1b6c0da80e (DEFAULT_TABLE table=test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260810 22:04:55.362730 32170 tablet_service.cc:1467] Processing CreateTablet for tablet f5b7bfea936e4001887b340805850f7a (DEFAULT_TABLE table=test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260810 22:04:55.367909 32168 tablet_service.cc:1467] Processing CreateTablet for tablet 26ac18eeb4a54ec2865d5bed0b18c84a (DEFAULT_TABLE table=test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260810 22:04:55.367915 31798 tablet_service.cc:1467] Processing CreateTablet for tablet f5b7bfea936e4001887b340805850f7a (DEFAULT_TABLE table=test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260810 22:04:55.370491 31795 tablet_service.cc:1467] Processing CreateTablet for tablet 79023ddd04404bc0ba3d3e673cd80c08 (DEFAULT_TABLE table=test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260810 22:04:55.372411 32166 tablet_service.cc:1467] Processing CreateTablet for tablet 79023ddd04404bc0ba3d3e673cd80c08 (DEFAULT_TABLE table=test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260810 22:04:55.378389 31797 tablet_service.cc:1467] Processing CreateTablet for tablet f4bc97c76ce24181bbac8c1b6c0da80e (DEFAULT_TABLE table=test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260810 22:04:55.382164 31796 tablet_service.cc:1467] Processing CreateTablet for tablet 26ac18eeb4a54ec2865d5bed0b18c84a (DEFAULT_TABLE table=test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260810 22:04:55.441038 32487 tablet_bootstrap.cc:492] T 0bac2f72b88b4c74a8ef33e484828638 P e07d7c5a69cf4e2e938b11c692e3565c: Bootstrap starting.
I20260810 22:04:55.442670 32488 tablet_bootstrap.cc:492] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c: Bootstrap starting.
I20260810 22:04:55.463129 32490 tablet_bootstrap.cc:492] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9: Bootstrap starting.
I20260810 22:04:55.467775 32487 tablet_bootstrap.cc:654] T 0bac2f72b88b4c74a8ef33e484828638 P e07d7c5a69cf4e2e938b11c692e3565c: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:55.473968 32489 tablet_bootstrap.cc:492] T 0bac2f72b88b4c74a8ef33e484828638 P 44c1b033bb9541a7bc66207ff2deba61: Bootstrap starting.
I20260810 22:04:55.474342 32498 tablet_bootstrap.cc:492] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c: Bootstrap starting.
I20260810 22:04:55.478823 32487 log.cc:824] T 0bac2f72b88b4c74a8ef33e484828638 P e07d7c5a69cf4e2e938b11c692e3565c: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:55.483785 32492 tablet_bootstrap.cc:492] T 26ac18eeb4a54ec2865d5bed0b18c84a P dd8ff3433b064de08a86dce7ab10aab9: Bootstrap starting.
I20260810 22:04:55.481199 32496 tablet_bootstrap.cc:492] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9: Bootstrap starting.
I20260810 22:04:55.486320 32497 tablet_bootstrap.cc:492] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61: Bootstrap starting.
I20260810 22:04:55.486694 32490 tablet_bootstrap.cc:654] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:55.489308 32490 log.cc:824] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:55.496528 32496 tablet_bootstrap.cc:654] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:55.498471 32489 tablet_bootstrap.cc:654] T 0bac2f72b88b4c74a8ef33e484828638 P 44c1b033bb9541a7bc66207ff2deba61: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:55.479583 32488 tablet_bootstrap.cc:654] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:55.500265 32500 tablet_bootstrap.cc:492] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22: Bootstrap starting.
I20260810 22:04:55.503064 32507 tablet_bootstrap.cc:492] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22: Bootstrap starting.
I20260810 22:04:55.507968 32497 tablet_bootstrap.cc:654] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:55.508076 32499 tablet_bootstrap.cc:492] T f5b7bfea936e4001887b340805850f7a P e07d7c5a69cf4e2e938b11c692e3565c: Bootstrap starting.
I20260810 22:04:55.513053 32503 tablet_bootstrap.cc:492] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22: Bootstrap starting.
I20260810 22:04:55.495806 32498 tablet_bootstrap.cc:654] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:55.515533 32492 tablet_bootstrap.cc:654] T 26ac18eeb4a54ec2865d5bed0b18c84a P dd8ff3433b064de08a86dce7ab10aab9: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:55.522395 32504 tablet_bootstrap.cc:492] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22: Bootstrap starting.
I20260810 22:04:55.499941 32502 tablet_bootstrap.cc:492] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61: Bootstrap starting.
I20260810 22:04:55.511010 32489 log.cc:824] T 0bac2f72b88b4c74a8ef33e484828638 P 44c1b033bb9541a7bc66207ff2deba61: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:55.525534 32488 tablet_bootstrap.cc:492] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c: No bootstrap required, opened a new log
I20260810 22:04:55.526085 32488 ts_tablet_manager.cc:1397] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c: Time spent bootstrapping tablet: real 0.084s user 0.013s sys 0.003s
I20260810 22:04:55.528784 32500 tablet_bootstrap.cc:654] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:55.530860 32497 tablet_bootstrap.cc:492] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61: No bootstrap required, opened a new log
I20260810 22:04:55.531554 32500 log.cc:824] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:55.532326 32497 ts_tablet_manager.cc:1397] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61: Time spent bootstrapping tablet: real 0.050s user 0.010s sys 0.011s
I20260810 22:04:55.538780 32499 tablet_bootstrap.cc:654] T f5b7bfea936e4001887b340805850f7a P e07d7c5a69cf4e2e938b11c692e3565c: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:55.540706 32487 tablet_bootstrap.cc:492] T 0bac2f72b88b4c74a8ef33e484828638 P e07d7c5a69cf4e2e938b11c692e3565c: No bootstrap required, opened a new log
I20260810 22:04:55.510555 32501 tablet_bootstrap.cc:492] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61: Bootstrap starting.
I20260810 22:04:55.541235 32487 ts_tablet_manager.cc:1397] T 0bac2f72b88b4c74a8ef33e484828638 P e07d7c5a69cf4e2e938b11c692e3565c: Time spent bootstrapping tablet: real 0.101s user 0.008s sys 0.009s
I20260810 22:04:55.542497 32490 tablet_bootstrap.cc:492] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9: No bootstrap required, opened a new log
I20260810 22:04:55.543028 32490 ts_tablet_manager.cc:1397] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9: Time spent bootstrapping tablet: real 0.081s user 0.010s sys 0.006s
I20260810 22:04:55.551250 32504 tablet_bootstrap.cc:654] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:55.553958 32498 tablet_bootstrap.cc:492] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c: No bootstrap required, opened a new log
I20260810 22:04:55.554445 32498 ts_tablet_manager.cc:1397] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c: Time spent bootstrapping tablet: real 0.081s user 0.013s sys 0.000s
I20260810 22:04:55.556808 32496 tablet_bootstrap.cc:492] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9: No bootstrap required, opened a new log
I20260810 22:04:55.556991 32502 tablet_bootstrap.cc:654] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:55.557307 32496 ts_tablet_manager.cc:1397] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9: Time spent bootstrapping tablet: real 0.077s user 0.010s sys 0.005s
I20260810 22:04:55.563251 32492 tablet_bootstrap.cc:492] T 26ac18eeb4a54ec2865d5bed0b18c84a P dd8ff3433b064de08a86dce7ab10aab9: No bootstrap required, opened a new log
I20260810 22:04:55.563709 32492 ts_tablet_manager.cc:1397] T 26ac18eeb4a54ec2865d5bed0b18c84a P dd8ff3433b064de08a86dce7ab10aab9: Time spent bootstrapping tablet: real 0.081s user 0.005s sys 0.011s
I20260810 22:04:55.566099 32499 tablet_bootstrap.cc:492] T f5b7bfea936e4001887b340805850f7a P e07d7c5a69cf4e2e938b11c692e3565c: No bootstrap required, opened a new log
I20260810 22:04:55.566584 32499 ts_tablet_manager.cc:1397] T f5b7bfea936e4001887b340805850f7a P e07d7c5a69cf4e2e938b11c692e3565c: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.004s
I20260810 22:04:55.559693 32503 tablet_bootstrap.cc:654] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:55.563395 32507 tablet_bootstrap.cc:654] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:55.562469 32489 tablet_bootstrap.cc:492] T 0bac2f72b88b4c74a8ef33e484828638 P 44c1b033bb9541a7bc66207ff2deba61: No bootstrap required, opened a new log
I20260810 22:04:55.572451 32489 ts_tablet_manager.cc:1397] T 0bac2f72b88b4c74a8ef33e484828638 P 44c1b033bb9541a7bc66207ff2deba61: Time spent bootstrapping tablet: real 0.099s user 0.015s sys 0.011s
I20260810 22:04:55.563412 32501 tablet_bootstrap.cc:654] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61: Neither blocks nor log segments found. Creating new log.
I20260810 22:04:55.588965 32500 tablet_bootstrap.cc:492] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22: No bootstrap required, opened a new log
I20260810 22:04:55.589576 32500 ts_tablet_manager.cc:1397] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22: Time spent bootstrapping tablet: real 0.090s user 0.039s sys 0.039s
I20260810 22:04:55.592837 32502 tablet_bootstrap.cc:492] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61: No bootstrap required, opened a new log
I20260810 22:04:55.594386 32502 ts_tablet_manager.cc:1397] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61: Time spent bootstrapping tablet: real 0.095s user 0.014s sys 0.003s
I20260810 22:04:55.598644 32499 raft_consensus.cc:348] T f5b7bfea936e4001887b340805850f7a P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:55.599458 32499 raft_consensus.cc:374] T f5b7bfea936e4001887b340805850f7a P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:55.599735 32499 raft_consensus.cc:729] T f5b7bfea936e4001887b340805850f7a P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07d7c5a69cf4e2e938b11c692e3565c, State: Initialized, Role: FOLLOWER
I20260810 22:04:55.600028 32507 tablet_bootstrap.cc:492] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22: No bootstrap required, opened a new log
I20260810 22:04:55.600445 32507 ts_tablet_manager.cc:1397] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22: Time spent bootstrapping tablet: real 0.098s user 0.000s sys 0.013s
I20260810 22:04:55.600543 32499 consensus_queue.cc:260] T f5b7bfea936e4001887b340805850f7a P e07d7c5a69cf4e2e938b11c692e3565c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:55.602530 32501 tablet_bootstrap.cc:492] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61: No bootstrap required, opened a new log
I20260810 22:04:55.603009 32501 ts_tablet_manager.cc:1397] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61: Time spent bootstrapping tablet: real 0.093s user 0.014s sys 0.003s
I20260810 22:04:55.604439 32504 tablet_bootstrap.cc:492] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22: No bootstrap required, opened a new log
I20260810 22:04:55.604328 32487 raft_consensus.cc:348] T 0bac2f72b88b4c74a8ef33e484828638 P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } }
I20260810 22:04:55.604840 32504 ts_tablet_manager.cc:1397] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22: Time spent bootstrapping tablet: real 0.104s user 0.009s sys 0.003s
I20260810 22:04:55.605026 32487 raft_consensus.cc:374] T 0bac2f72b88b4c74a8ef33e484828638 P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:55.601852 32488 raft_consensus.cc:348] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:55.605314 32487 raft_consensus.cc:729] T 0bac2f72b88b4c74a8ef33e484828638 P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07d7c5a69cf4e2e938b11c692e3565c, State: Initialized, Role: FOLLOWER
I20260810 22:04:55.606153 32487 consensus_queue.cc:260] T 0bac2f72b88b4c74a8ef33e484828638 P e07d7c5a69cf4e2e938b11c692e3565c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } }
I20260810 22:04:55.606093 32497 raft_consensus.cc:348] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } }
I20260810 22:04:55.606375 32501 raft_consensus.cc:348] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:55.608273 32501 raft_consensus.cc:374] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:55.608716 32501 raft_consensus.cc:729] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c1b033bb9541a7bc66207ff2deba61, State: Initialized, Role: FOLLOWER
I20260810 22:04:55.605445 32488 raft_consensus.cc:374] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:55.610390 32488 raft_consensus.cc:729] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07d7c5a69cf4e2e938b11c692e3565c, State: Initialized, Role: FOLLOWER
I20260810 22:04:55.611290 32488 consensus_queue.cc:260] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:55.612457 32497 raft_consensus.cc:374] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:55.612809 32497 raft_consensus.cc:729] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c1b033bb9541a7bc66207ff2deba61, State: Initialized, Role: FOLLOWER
I20260810 22:04:55.607352 32489 raft_consensus.cc:348] T 0bac2f72b88b4c74a8ef33e484828638 P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } }
I20260810 22:04:55.614326 32489 raft_consensus.cc:374] T 0bac2f72b88b4c74a8ef33e484828638 P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:55.614948 32488 ts_tablet_manager.cc:1428] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c: Time spent starting tablet: real 0.089s user 0.017s sys 0.004s
I20260810 22:04:55.615283 32489 raft_consensus.cc:729] T 0bac2f72b88b4c74a8ef33e484828638 P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c1b033bb9541a7bc66207ff2deba61, State: Initialized, Role: FOLLOWER
I20260810 22:04:55.611619 32501 consensus_queue.cc:260] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:55.616266 32489 consensus_queue.cc:260] T 0bac2f72b88b4c74a8ef33e484828638 P 44c1b033bb9541a7bc66207ff2deba61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } }
I20260810 22:04:55.614729 32497 consensus_queue.cc:260] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } }
I20260810 22:04:55.618856 32502 raft_consensus.cc:348] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:55.619736 32502 raft_consensus.cc:374] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:55.620071 32502 raft_consensus.cc:729] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c1b033bb9541a7bc66207ff2deba61, State: Initialized, Role: FOLLOWER
I20260810 22:04:55.620798 32499 ts_tablet_manager.cc:1428] T f5b7bfea936e4001887b340805850f7a P e07d7c5a69cf4e2e938b11c692e3565c: Time spent starting tablet: real 0.054s user 0.007s sys 0.002s
I20260810 22:04:55.620822 32502 consensus_queue.cc:260] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:55.621641 32498 raft_consensus.cc:348] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } }
I20260810 22:04:55.622359 32498 raft_consensus.cc:374] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:55.622642 32498 raft_consensus.cc:729] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07d7c5a69cf4e2e938b11c692e3565c, State: Initialized, Role: FOLLOWER
I20260810 22:04:55.623272 32498 consensus_queue.cc:260] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } }
I20260810 22:04:55.629120 32497 ts_tablet_manager.cc:1428] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61: Time spent starting tablet: real 0.096s user 0.022s sys 0.011s
I20260810 22:04:55.629434 32501 ts_tablet_manager.cc:1428] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61: Time spent starting tablet: real 0.026s user 0.008s sys 0.003s
I20260810 22:04:55.633291 32489 ts_tablet_manager.cc:1428] T 0bac2f72b88b4c74a8ef33e484828638 P 44c1b033bb9541a7bc66207ff2deba61: Time spent starting tablet: real 0.060s user 0.015s sys 0.007s
I20260810 22:04:55.635350 32498 ts_tablet_manager.cc:1428] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c: Time spent starting tablet: real 0.081s user 0.013s sys 0.000s
I20260810 22:04:55.636279 32502 ts_tablet_manager.cc:1428] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61: Time spent starting tablet: real 0.042s user 0.005s sys 0.007s
I20260810 22:04:55.639412 32487 ts_tablet_manager.cc:1428] T 0bac2f72b88b4c74a8ef33e484828638 P e07d7c5a69cf4e2e938b11c692e3565c: Time spent starting tablet: real 0.098s user 0.013s sys 0.006s
I20260810 22:04:55.649211 32527 raft_consensus.cc:482] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:55.649827 32527 raft_consensus.cc:504] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } }
I20260810 22:04:55.653280 32496 raft_consensus.cc:348] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } }
I20260810 22:04:55.655103 32496 raft_consensus.cc:374] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:55.655472 32496 raft_consensus.cc:729] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8ff3433b064de08a86dce7ab10aab9, State: Initialized, Role: FOLLOWER
I20260810 22:04:55.657822 32490 raft_consensus.cc:348] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } }
I20260810 22:04:55.658545 32490 raft_consensus.cc:374] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:55.658818 32490 raft_consensus.cc:729] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8ff3433b064de08a86dce7ab10aab9, State: Initialized, Role: FOLLOWER
I20260810 22:04:55.659613 32490 consensus_queue.cc:260] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } }
I20260810 22:04:55.660171 32503 tablet_bootstrap.cc:492] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22: No bootstrap required, opened a new log
I20260810 22:04:55.660632 32503 ts_tablet_manager.cc:1397] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22: Time spent bootstrapping tablet: real 0.148s user 0.012s sys 0.012s
I20260810 22:04:55.661202 32527 leader_election.cc:288] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103), dd8ff3433b064de08a86dce7ab10aab9 (127.27.17.132:34125)
I20260810 22:04:55.662232 32492 raft_consensus.cc:348] T 26ac18eeb4a54ec2865d5bed0b18c84a P dd8ff3433b064de08a86dce7ab10aab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } }
I20260810 22:04:55.663162 32492 raft_consensus.cc:374] T 26ac18eeb4a54ec2865d5bed0b18c84a P dd8ff3433b064de08a86dce7ab10aab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:55.663501 32492 raft_consensus.cc:729] T 26ac18eeb4a54ec2865d5bed0b18c84a P dd8ff3433b064de08a86dce7ab10aab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8ff3433b064de08a86dce7ab10aab9, State: Initialized, Role: FOLLOWER
I20260810 22:04:55.664327 32492 consensus_queue.cc:260] T 26ac18eeb4a54ec2865d5bed0b18c84a P dd8ff3433b064de08a86dce7ab10aab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } }
I20260810 22:04:55.660816 32496 consensus_queue.cc:260] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } }
I20260810 22:04:55.671904 32453 heartbeater.cc:503] Master 127.27.17.190:44707 was elected leader, sending a full tablet report...
I20260810 22:04:55.672713 32504 raft_consensus.cc:348] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } }
I20260810 22:04:55.673813 32504 raft_consensus.cc:374] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:55.675236 32504 raft_consensus.cc:729] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee255d32da241aeb929742ee6005b22, State: Initialized, Role: FOLLOWER
I20260810 22:04:55.676085 32492 ts_tablet_manager.cc:1428] T 26ac18eeb4a54ec2865d5bed0b18c84a P dd8ff3433b064de08a86dce7ab10aab9: Time spent starting tablet: real 0.112s user 0.013s sys 0.001s
I20260810 22:04:55.674507 32503 raft_consensus.cc:348] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } }
I20260810 22:04:55.676786 32500 raft_consensus.cc:348] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:55.676972 32503 raft_consensus.cc:374] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:55.679230 32503 raft_consensus.cc:729] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee255d32da241aeb929742ee6005b22, State: Initialized, Role: FOLLOWER
I20260810 22:04:55.676332 32504 consensus_queue.cc:260] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } }
I20260810 22:04:55.678885 32500 raft_consensus.cc:374] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:55.681608 32500 raft_consensus.cc:729] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee255d32da241aeb929742ee6005b22, State: Initialized, Role: FOLLOWER
I20260810 22:04:55.680006 32503 consensus_queue.cc:260] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } }
I20260810 22:04:55.686949 32490 ts_tablet_manager.cc:1428] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9: Time spent starting tablet: real 0.144s user 0.025s sys 0.014s
I20260810 22:04:55.687752 32500 consensus_queue.cc:260] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:55.690413 31825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ac18eeb4a54ec2865d5bed0b18c84a" candidate_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee255d32da241aeb929742ee6005b22" is_pre_election: true
I20260810 22:04:55.691468 32504 ts_tablet_manager.cc:1428] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22: Time spent starting tablet: real 0.086s user 0.006s sys 0.003s
I20260810 22:04:55.691363 31825 raft_consensus.cc:2454] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07d7c5a69cf4e2e938b11c692e3565c in term 0.
I20260810 22:04:55.692762 32110 leader_election.cc:302] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dee255d32da241aeb929742ee6005b22, e07d7c5a69cf4e2e938b11c692e3565c; no voters:
I20260810 22:04:55.693252 32503 ts_tablet_manager.cc:1428] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22: Time spent starting tablet: real 0.032s user 0.003s sys 0.005s
I20260810 22:04:55.693917 32527 raft_consensus.cc:2781] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:55.694293 32527 raft_consensus.cc:482] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:55.694633 32527 raft_consensus.cc:3037] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:55.695063 32500 ts_tablet_manager.cc:1428] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22: Time spent starting tablet: real 0.105s user 0.016s sys 0.014s
W20260810 22:04:55.696107 32257 tablet.cc:2369] T 0bac2f72b88b4c74a8ef33e484828638 P e07d7c5a69cf4e2e938b11c692e3565c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:04:55.696024 32507 raft_consensus.cc:348] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:55.696782 32507 raft_consensus.cc:374] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:04:55.697059 32507 raft_consensus.cc:729] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee255d32da241aeb929742ee6005b22, State: Initialized, Role: FOLLOWER
I20260810 22:04:55.697747 32507 consensus_queue.cc:260] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:55.699853 32507 ts_tablet_manager.cc:1428] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22: Time spent starting tablet: real 0.099s user 0.010s sys 0.004s
I20260810 22:04:55.701797 32527 raft_consensus.cc:504] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } }
I20260810 22:04:55.704538 32527 leader_election.cc:288] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c [CANDIDATE]: Term 1 election: Requested vote from peers dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103), dd8ff3433b064de08a86dce7ab10aab9 (127.27.17.132:34125)
I20260810 22:04:55.705009 31825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ac18eeb4a54ec2865d5bed0b18c84a" candidate_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee255d32da241aeb929742ee6005b22"
I20260810 22:04:55.705264 32496 ts_tablet_manager.cc:1428] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9: Time spent starting tablet: real 0.148s user 0.009s sys 0.014s
I20260810 22:04:55.705561 31825 raft_consensus.cc:3037] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:55.711334 32534 raft_consensus.cc:482] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:55.711570 32530 raft_consensus.cc:482] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:55.711943 32534 raft_consensus.cc:504] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:55.712162 32530 raft_consensus.cc:504] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } }
I20260810 22:04:55.713256 32540 raft_consensus.cc:482] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:55.713815 32540 raft_consensus.cc:504] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } }
I20260810 22:04:55.714811 31825 raft_consensus.cc:2454] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07d7c5a69cf4e2e938b11c692e3565c in term 1.
I20260810 22:04:55.715751 32530 leader_election.cc:288] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103), dd8ff3433b064de08a86dce7ab10aab9 (127.27.17.132:34125)
I20260810 22:04:55.715955 32110 leader_election.cc:302] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dee255d32da241aeb929742ee6005b22, e07d7c5a69cf4e2e938b11c692e3565c; no voters:
I20260810 22:04:55.716820 32527 raft_consensus.cc:2781] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:55.717682 32527 raft_consensus.cc:686] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c [term 1 LEADER]: Becoming Leader. State: Replica: e07d7c5a69cf4e2e938b11c692e3565c, State: Running, Role: LEADER
I20260810 22:04:55.719699 32534 leader_election.cc:288] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103), e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131:38509)
I20260810 22:04:55.721282 32381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ac18eeb4a54ec2865d5bed0b18c84a" candidate_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8ff3433b064de08a86dce7ab10aab9" is_pre_election: true
I20260810 22:04:55.719069 32527 consensus_queue.cc:237] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } }
I20260810 22:04:55.722149 32381 raft_consensus.cc:2454] T 26ac18eeb4a54ec2865d5bed0b18c84a P dd8ff3433b064de08a86dce7ab10aab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07d7c5a69cf4e2e938b11c692e3565c in term 0.
I20260810 22:04:55.723747 32380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ac18eeb4a54ec2865d5bed0b18c84a" candidate_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8ff3433b064de08a86dce7ab10aab9"
I20260810 22:04:55.724591 32380 raft_consensus.cc:3037] T 26ac18eeb4a54ec2865d5bed0b18c84a P dd8ff3433b064de08a86dce7ab10aab9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:55.739418 32380 raft_consensus.cc:2454] T 26ac18eeb4a54ec2865d5bed0b18c84a P dd8ff3433b064de08a86dce7ab10aab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07d7c5a69cf4e2e938b11c692e3565c in term 1.
I20260810 22:04:55.744385 32540 leader_election.cc:288] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131:38509), 44c1b033bb9541a7bc66207ff2deba61 (127.27.17.130:46865)
I20260810 22:04:55.747822 31606 catalog_manager.cc:5495] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c reported cstate change: term changed from 0 to 1, leader changed from <none> to e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131). New cstate: current_term: 1 leader_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } health_report { overall_health: UNKNOWN } } }
I20260810 22:04:55.770349 32380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bc97c76ce24181bbac8c1b6c0da80e" candidate_uuid: "44c1b033bb9541a7bc66207ff2deba61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8ff3433b064de08a86dce7ab10aab9" is_pre_election: true
I20260810 22:04:55.771021 32380 raft_consensus.cc:2454] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c1b033bb9541a7bc66207ff2deba61 in term 0.
I20260810 22:04:55.772346 31930 leader_election.cc:302] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44c1b033bb9541a7bc66207ff2deba61, dd8ff3433b064de08a86dce7ab10aab9; no voters:
I20260810 22:04:55.773456 32534 raft_consensus.cc:2781] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:55.773842 32534 raft_consensus.cc:482] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:55.774190 32534 raft_consensus.cc:3037] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:55.773974 31825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b7bfea936e4001887b340805850f7a" candidate_uuid: "44c1b033bb9541a7bc66207ff2deba61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee255d32da241aeb929742ee6005b22" is_pre_election: true
I20260810 22:04:55.774757 31825 raft_consensus.cc:2454] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c1b033bb9541a7bc66207ff2deba61 in term 0.
I20260810 22:04:55.776095 31824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bc97c76ce24181bbac8c1b6c0da80e" candidate_uuid: "44c1b033bb9541a7bc66207ff2deba61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee255d32da241aeb929742ee6005b22" is_pre_election: true
I20260810 22:04:55.776983 31824 raft_consensus.cc:2454] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c1b033bb9541a7bc66207ff2deba61 in term 0.
I20260810 22:04:55.779059 31926 leader_election.cc:302] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44c1b033bb9541a7bc66207ff2deba61, dee255d32da241aeb929742ee6005b22; no voters:
I20260810 22:04:55.780716 32530 raft_consensus.cc:2781] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:55.781075 32530 raft_consensus.cc:482] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:55.781509 32530 raft_consensus.cc:3037] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:55.782107 32534 raft_consensus.cc:504] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } }
I20260810 22:04:55.784907 31824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bc97c76ce24181bbac8c1b6c0da80e" candidate_uuid: "44c1b033bb9541a7bc66207ff2deba61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee255d32da241aeb929742ee6005b22"
I20260810 22:04:55.785542 31824 raft_consensus.cc:3037] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:55.787953 32195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b7bfea936e4001887b340805850f7a" candidate_uuid: "44c1b033bb9541a7bc66207ff2deba61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" is_pre_election: true
I20260810 22:04:55.788830 32195 raft_consensus.cc:2454] T f5b7bfea936e4001887b340805850f7a P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c1b033bb9541a7bc66207ff2deba61 in term 0.
I20260810 22:04:55.789028 32380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bc97c76ce24181bbac8c1b6c0da80e" candidate_uuid: "44c1b033bb9541a7bc66207ff2deba61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8ff3433b064de08a86dce7ab10aab9"
I20260810 22:04:55.789614 32380 raft_consensus.cc:3037] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:55.791772 32195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bac2f72b88b4c74a8ef33e484828638" candidate_uuid: "dd8ff3433b064de08a86dce7ab10aab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" is_pre_election: true
I20260810 22:04:55.792358 32195 raft_consensus.cc:2454] T 0bac2f72b88b4c74a8ef33e484828638 P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd8ff3433b064de08a86dce7ab10aab9 in term 0.
I20260810 22:04:55.793435 32534 leader_election.cc:288] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [CANDIDATE]: Term 1 election: Requested vote from peers dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103), dd8ff3433b064de08a86dce7ab10aab9 (127.27.17.132:34125)
I20260810 22:04:55.794078 32293 leader_election.cc:302] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd8ff3433b064de08a86dce7ab10aab9, e07d7c5a69cf4e2e938b11c692e3565c; no voters:
I20260810 22:04:55.795178 32540 raft_consensus.cc:2781] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:55.795524 32540 raft_consensus.cc:482] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:55.795853 32540 raft_consensus.cc:3037] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:55.796640 32380 raft_consensus.cc:2454] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c1b033bb9541a7bc66207ff2deba61 in term 1.
I20260810 22:04:55.797695 31930 leader_election.cc:302] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44c1b033bb9541a7bc66207ff2deba61, dd8ff3433b064de08a86dce7ab10aab9; no voters:
I20260810 22:04:55.797530 32530 raft_consensus.cc:504] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:55.798550 32534 raft_consensus.cc:2781] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:55.799027 32534 raft_consensus.cc:686] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [term 1 LEADER]: Becoming Leader. State: Replica: 44c1b033bb9541a7bc66207ff2deba61, State: Running, Role: LEADER
I20260810 22:04:55.800899 32530 leader_election.cc:288] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 [CANDIDATE]: Term 1 election: Requested vote from peers dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103), e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131:38509)
I20260810 22:04:55.801141 31824 raft_consensus.cc:2454] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c1b033bb9541a7bc66207ff2deba61 in term 1.
I20260810 22:04:55.799933 32534 consensus_queue.cc:237] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } }
I20260810 22:04:55.801820 31825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b7bfea936e4001887b340805850f7a" candidate_uuid: "44c1b033bb9541a7bc66207ff2deba61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee255d32da241aeb929742ee6005b22"
I20260810 22:04:55.802376 31825 raft_consensus.cc:3037] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:55.803656 32540 raft_consensus.cc:504] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } }
I20260810 22:04:55.805655 32195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b7bfea936e4001887b340805850f7a" candidate_uuid: "44c1b033bb9541a7bc66207ff2deba61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07d7c5a69cf4e2e938b11c692e3565c"
I20260810 22:04:55.806218 32195 raft_consensus.cc:3037] T f5b7bfea936e4001887b340805850f7a P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:55.810913 31825 raft_consensus.cc:2454] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c1b033bb9541a7bc66207ff2deba61 in term 1.
I20260810 22:04:55.812237 31926 leader_election.cc:302] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44c1b033bb9541a7bc66207ff2deba61, dee255d32da241aeb929742ee6005b22; no voters:
I20260810 22:04:55.813380 32530 raft_consensus.cc:2781] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 [term 1 FOLLOWER]: Leader election won for term 1
W20260810 22:04:55.814200 32074 tablet.cc:2369] T 0bac2f72b88b4c74a8ef33e484828638 P 44c1b033bb9541a7bc66207ff2deba61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:04:55.814231 32530 raft_consensus.cc:686] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 [term 1 LEADER]: Becoming Leader. State: Replica: 44c1b033bb9541a7bc66207ff2deba61, State: Running, Role: LEADER
I20260810 22:04:55.815645 32017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bac2f72b88b4c74a8ef33e484828638" candidate_uuid: "dd8ff3433b064de08a86dce7ab10aab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c1b033bb9541a7bc66207ff2deba61" is_pre_election: true
I20260810 22:04:55.815424 32530 consensus_queue.cc:237] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:55.816392 32017 raft_consensus.cc:2454] T 0bac2f72b88b4c74a8ef33e484828638 P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd8ff3433b064de08a86dce7ab10aab9 in term 0.
I20260810 22:04:55.817695 32541 raft_consensus.cc:482] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:55.818194 32541 raft_consensus.cc:504] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:55.822947 32541 leader_election.cc:288] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44c1b033bb9541a7bc66207ff2deba61 (127.27.17.130:46865), e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131:38509)
I20260810 22:04:55.827304 32540 leader_election.cc:288] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 [CANDIDATE]: Term 1 election: Requested vote from peers e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131:38509), 44c1b033bb9541a7bc66207ff2deba61 (127.27.17.130:46865)
I20260810 22:04:55.829871 32194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bac2f72b88b4c74a8ef33e484828638" candidate_uuid: "dd8ff3433b064de08a86dce7ab10aab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07d7c5a69cf4e2e938b11c692e3565c"
I20260810 22:04:55.830785 32194 raft_consensus.cc:3037] T 0bac2f72b88b4c74a8ef33e484828638 P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:55.831184 32195 raft_consensus.cc:2454] T f5b7bfea936e4001887b340805850f7a P e07d7c5a69cf4e2e938b11c692e3565c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c1b033bb9541a7bc66207ff2deba61 in term 1.
I20260810 22:04:55.836040 32017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bac2f72b88b4c74a8ef33e484828638" candidate_uuid: "dd8ff3433b064de08a86dce7ab10aab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c1b033bb9541a7bc66207ff2deba61"
I20260810 22:04:55.836596 32017 raft_consensus.cc:3037] T 0bac2f72b88b4c74a8ef33e484828638 P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:55.844267 32017 raft_consensus.cc:2454] T 0bac2f72b88b4c74a8ef33e484828638 P 44c1b033bb9541a7bc66207ff2deba61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8ff3433b064de08a86dce7ab10aab9 in term 1.
I20260810 22:04:55.847411 32194 raft_consensus.cc:2454] T 0bac2f72b88b4c74a8ef33e484828638 P e07d7c5a69cf4e2e938b11c692e3565c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8ff3433b064de08a86dce7ab10aab9 in term 1.
I20260810 22:04:55.854502 32291 leader_election.cc:302] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44c1b033bb9541a7bc66207ff2deba61, dd8ff3433b064de08a86dce7ab10aab9; no voters:
I20260810 22:04:55.855558 32540 raft_consensus.cc:2781] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:55.856058 32540 raft_consensus.cc:686] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 [term 1 LEADER]: Becoming Leader. State: Replica: dd8ff3433b064de08a86dce7ab10aab9, State: Running, Role: LEADER
I20260810 22:04:55.857005 32540 consensus_queue.cc:237] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } }
W20260810 22:04:55.864763 32454 tablet.cc:2369] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:04:55.868847 32017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79023ddd04404bc0ba3d3e673cd80c08" candidate_uuid: "dee255d32da241aeb929742ee6005b22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c1b033bb9541a7bc66207ff2deba61" is_pre_election: true
I20260810 22:04:55.870497 32017 raft_consensus.cc:2454] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dee255d32da241aeb929742ee6005b22 in term 0.
I20260810 22:04:55.870605 31606 catalog_manager.cc:5495] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44c1b033bb9541a7bc66207ff2deba61 (127.27.17.130). New cstate: current_term: 1 leader_uuid: "44c1b033bb9541a7bc66207ff2deba61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } health_report { overall_health: HEALTHY } } }
I20260810 22:04:55.871874 31729 leader_election.cc:302] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44c1b033bb9541a7bc66207ff2deba61, dee255d32da241aeb929742ee6005b22; no voters:
I20260810 22:04:55.873114 32541 raft_consensus.cc:2781] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:04:55.873517 32541 raft_consensus.cc:482] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:55.873847 32541 raft_consensus.cc:3037] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:55.881826 32541 raft_consensus.cc:504] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:55.882040 31607 catalog_manager.cc:5495] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd8ff3433b064de08a86dce7ab10aab9 (127.27.17.132). New cstate: current_term: 1 leader_uuid: "dd8ff3433b064de08a86dce7ab10aab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } health_report { overall_health: UNKNOWN } } }
I20260810 22:04:55.879082 32194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79023ddd04404bc0ba3d3e673cd80c08" candidate_uuid: "dee255d32da241aeb929742ee6005b22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" is_pre_election: true
I20260810 22:04:55.884022 32541 leader_election.cc:288] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 [CANDIDATE]: Term 1 election: Requested vote from peers 44c1b033bb9541a7bc66207ff2deba61 (127.27.17.130:46865), e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131:38509)
I20260810 22:04:55.884369 32194 raft_consensus.cc:2454] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dee255d32da241aeb929742ee6005b22 in term 0.
I20260810 22:04:55.884800 32017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79023ddd04404bc0ba3d3e673cd80c08" candidate_uuid: "dee255d32da241aeb929742ee6005b22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c1b033bb9541a7bc66207ff2deba61"
I20260810 22:04:55.885121 32195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79023ddd04404bc0ba3d3e673cd80c08" candidate_uuid: "dee255d32da241aeb929742ee6005b22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07d7c5a69cf4e2e938b11c692e3565c"
I20260810 22:04:55.885452 32017 raft_consensus.cc:3037] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:04:55.885758 32195 raft_consensus.cc:3037] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [term 0 FOLLOWER]: Advancing to term 1
W20260810 22:04:55.888518 31890 tablet.cc:2369] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:04:55.891861 32017 raft_consensus.cc:2454] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee255d32da241aeb929742ee6005b22 in term 1.
I20260810 22:04:55.893006 31729 leader_election.cc:302] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44c1b033bb9541a7bc66207ff2deba61, dee255d32da241aeb929742ee6005b22; no voters:
I20260810 22:04:55.892489 31605 catalog_manager.cc:5495] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44c1b033bb9541a7bc66207ff2deba61 (127.27.17.130). New cstate: current_term: 1 leader_uuid: "44c1b033bb9541a7bc66207ff2deba61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } health_report { overall_health: UNKNOWN } } }
I20260810 22:04:55.895188 32195 raft_consensus.cc:2454] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee255d32da241aeb929742ee6005b22 in term 1.
I20260810 22:04:55.895426 32541 raft_consensus.cc:2781] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:04:55.896111 32541 raft_consensus.cc:686] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 [term 1 LEADER]: Becoming Leader. State: Replica: dee255d32da241aeb929742ee6005b22, State: Running, Role: LEADER
I20260810 22:04:55.897032 32541 consensus_queue.cc:237] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:55.907810 31605 catalog_manager.cc:5495] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 reported cstate change: term changed from 0 to 1, leader changed from <none> to dee255d32da241aeb929742ee6005b22 (127.27.17.129). New cstate: current_term: 1 leader_uuid: "dee255d32da241aeb929742ee6005b22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } health_report { overall_health: UNKNOWN } } }
I20260810 22:04:56.003410 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 31694
I20260810 22:04:56.047369 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.129:34103
--local_ip_for_outbound_sockets=127.27.17.129
--tserver_master_addrs=127.27.17.190:44707
--webserver_port=38519
--webserver_interface=127.27.17.129
--env_inject_eio_globs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-1/**
--env_inject_eio=1.0
--crash_on_eio=false
--builtin_ntp_servers=127.27.17.148:38205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260810 22:04:56.254877 32534 consensus_queue.cc:1035] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:04:56.266489 32530 consensus_queue.cc:1035] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 22:04:56.282029 31926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.129:34103: connect: Connection refused (error 111)
I20260810 22:04:56.288362 32572 consensus_queue.cc:1035] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 22:04:56.297163 32110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.129:34103: connect: Connection refused (error 111)
W20260810 22:04:56.316087 31926 consensus_peers.cc:487] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 -> Peer dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103): Couldn't send request to peer dee255d32da241aeb929742ee6005b22. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:34103: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 22:04:56.330384 32110 consensus_peers.cc:487] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c -> Peer dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103): Couldn't send request to peer dee255d32da241aeb929742ee6005b22. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:34103: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 22:04:56.363323 31926 consensus_peers.cc:487] T f5b7bfea936e4001887b340805850f7a P 44c1b033bb9541a7bc66207ff2deba61 -> Peer dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103): Couldn't send request to peer dee255d32da241aeb929742ee6005b22. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:34103: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:56.392985 32540 consensus_queue.cc:1035] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 22:04:56.418414 32540 consensus_queue.cc:1035] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 22:04:56.459331 32568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:04:56.459825 32568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:04:56.460630 32568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:04:56.492298 32568 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20260810 22:04:56.492810 32568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:04:56.493945 32568 flags.cc:425] Enabled experimental flag: --crash_on_eio=false
W20260810 22:04:56.494573 32568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.129
I20260810 22:04:56.529279 32568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:38205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.129:34103
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.27.17.129
--webserver_port=38519
--tserver_master_addrs=127.27.17.190:44707
--env_inject_eio=1
--env_inject_eio_globs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-1/**
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.129
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:04:56.530615 32568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260810 22:04:56.531621 32568 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260810 22:04:56.531843 32568 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260810 22:04:56.533062 32568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:04:56.554224 32604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:04:57.555490 32576 raft_consensus.cc:482] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:04:57.555949 32576 raft_consensus.cc:504] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:57.558108 32576 leader_election.cc:288] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103), 44c1b033bb9541a7bc66207ff2deba61 (127.27.17.130:46865)
W20260810 22:04:57.565039 32110 leader_election.cc:334] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103): Network error: Client connection negotiation failed: client connection to 127.27.17.129:34103: connect: Connection refused (error 111)
I20260810 22:04:57.573581 32017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79023ddd04404bc0ba3d3e673cd80c08" candidate_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c1b033bb9541a7bc66207ff2deba61" is_pre_election: true
I20260810 22:04:57.574536 32017 raft_consensus.cc:2454] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07d7c5a69cf4e2e938b11c692e3565c in term 1.
I20260810 22:04:57.575846 32110 leader_election.cc:302] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44c1b033bb9541a7bc66207ff2deba61, e07d7c5a69cf4e2e938b11c692e3565c; no voters: dee255d32da241aeb929742ee6005b22
I20260810 22:04:57.576654 32576 raft_consensus.cc:2781] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 22:04:57.577157 32576 raft_consensus.cc:482] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:04:57.577630 32576 raft_consensus.cc:3037] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:04:57.585788 32576 raft_consensus.cc:504] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:57.587857 32576 leader_election.cc:288] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [CANDIDATE]: Term 2 election: Requested vote from peers dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103), 44c1b033bb9541a7bc66207ff2deba61 (127.27.17.130:46865)
I20260810 22:04:57.589640 32017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79023ddd04404bc0ba3d3e673cd80c08" candidate_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c1b033bb9541a7bc66207ff2deba61"
I20260810 22:04:57.590217 32017 raft_consensus.cc:3037] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61 [term 1 FOLLOWER]: Advancing to term 2
W20260810 22:04:57.590835 32110 leader_election.cc:334] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103): Network error: Client connection negotiation failed: client connection to 127.27.17.129:34103: connect: Connection refused (error 111)
I20260810 22:04:57.596793 32017 raft_consensus.cc:2454] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07d7c5a69cf4e2e938b11c692e3565c in term 2.
I20260810 22:04:57.597823 32110 leader_election.cc:302] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44c1b033bb9541a7bc66207ff2deba61, e07d7c5a69cf4e2e938b11c692e3565c; no voters: dee255d32da241aeb929742ee6005b22
I20260810 22:04:57.598575 32576 raft_consensus.cc:2781] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:04:57.599593 32576 raft_consensus.cc:686] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [term 2 LEADER]: Becoming Leader. State: Replica: e07d7c5a69cf4e2e938b11c692e3565c, State: Running, Role: LEADER
I20260810 22:04:57.600329 32576 consensus_queue.cc:237] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:57.607240 31605 catalog_manager.cc:5495] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c reported cstate change: term changed from 1 to 2, leader changed from dee255d32da241aeb929742ee6005b22 (127.27.17.129) to e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131). New cstate: current_term: 2 leader_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } health_report { overall_health: HEALTHY } } }
W20260810 22:04:56.554216 32601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:56.555495 32602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:04:57.795056 32603 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 22:04:57.795192 32568 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 22:04:57.799551 32568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:04:57.802515 32568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:04:57.803954 32568 hybrid_clock.cc:648] HybridClock initialized: now 1786399497803927 us; error 49 us; skew 500 ppm
I20260810 22:04:57.804754 32568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:04:57.812757 32568 webserver.cc:460] Webserver started at http://127.27.17.129:38519/ using document root <none> and password file <none>
I20260810 22:04:57.814548 32568 fs_manager.cc:359] Metadata directory not provided
I20260810 22:04:57.814764 32568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
E20260810 22:04:57.817384 32568 env_posix.cc:432] I/O error, context: INJECTED FAILURE
E20260810 22:04:57.822377 32568 env_posix.cc:432] I/O error, context: INJECTED FAILURE
E20260810 22:04:57.838896 32568 env_posix.cc:432] I/O error, context: INJECTED FAILURE
I20260810 22:04:57.852178 32568 fs_manager.cc:709] Time spent opening directory manager: real 0.031s user 0.023s sys 0.010s
E20260810 22:04:57.857560 32568 log_block_manager.cc:2924] Not using report from /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-1/data: IO error: Data directory failed (error 5)
I20260810 22:04:57.857954 32630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:57.858494 32630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:57.859928 32630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:57.860100 32631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:04:57.861750 32568 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.004s
I20260810 22:04:57.862145 32568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/wal
uuid: "dee255d32da241aeb929742ee6005b22"
format_stamp: "Formatted at 2026-08-10 22:04:49 on dist-test-slave-bw88"
I20260810 22:04:57.862730 32568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/wal
4 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:04:57.906369 32568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:04:57.907979 32568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:04:57.908527 32568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:04:57.911443 32568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:04:57.919003 32641 ts_tablet_manager.cc:542] Loading tablet metadata (0/4 complete)
W20260810 22:04:57.932466 32645 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet 79023ddd04404bc0ba3d3e673cd80c08: could not find data dir with uuid 2bfab9e5e5814428a7f78f599cd99d3f
W20260810 22:04:57.933477 32646 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet f4bc97c76ce24181bbac8c1b6c0da80e: could not find data dir with uuid 2bfab9e5e5814428a7f78f599cd99d3f
I20260810 22:04:57.934196 32568 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20260810 22:04:57.934437 32568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.018s user 0.000s sys 0.008s
I20260810 22:04:57.934733 32568 ts_tablet_manager.cc:594] Registering tablets (0/4 complete)
I20260810 22:04:57.940353 32646 tablet_bootstrap.cc:492] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22: Bootstrap starting.
I20260810 22:04:57.944449 32641 tablet_bootstrap.cc:492] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22: Bootstrap starting.
I20260810 22:04:57.950098 32645 tablet_bootstrap.cc:492] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22: Bootstrap starting.
E20260810 22:04:57.950624 32645 ts_tablet_manager.cc:1419] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 79023ddd04404bc0ba3d3e673cd80c08
I20260810 22:04:57.950939 32645 ts_tablet_manager.cc:1397] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:04:57.951329 32645 tablet_replica.cc:330] stopping tablet replica
I20260810 22:04:57.951658 32645 raft_consensus.cc:2229] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:04:57.955330 32645 raft_consensus.cc:2258] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:04:57.955819 32568 ts_tablet_manager.cc:610] Registered 4 tablets
I20260810 22:04:57.956063 32568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.021s user 0.019s sys 0.000s
I20260810 22:04:57.956210 32645 tablet_bootstrap.cc:492] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22: Bootstrap starting.
E20260810 22:04:57.956810 32645 ts_tablet_manager.cc:1419] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet f4bc97c76ce24181bbac8c1b6c0da80e
I20260810 22:04:57.957165 32645 ts_tablet_manager.cc:1397] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:04:57.957477 32645 tablet_replica.cc:330] stopping tablet replica
I20260810 22:04:57.957770 32645 raft_consensus.cc:2229] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:04:57.959817 32645 raft_consensus.cc:2258] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:04:58.006399 32641 log.cc:824] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22: Log is configured to *not* fsync() on all Append() calls
I20260810 22:04:58.015329 32641 tablet_bootstrap.cc:492] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:04:58.016628 32641 tablet_bootstrap.cc:492] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22: Bootstrap complete.
I20260810 22:04:58.018675 32641 ts_tablet_manager.cc:1397] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22: Time spent bootstrapping tablet: real 0.074s user 0.034s sys 0.031s
I20260810 22:04:58.018862 32646 tablet_bootstrap.cc:492] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:04:58.019905 32646 tablet_bootstrap.cc:492] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22: Bootstrap complete.
I20260810 22:04:58.022118 32646 ts_tablet_manager.cc:1397] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22: Time spent bootstrapping tablet: real 0.082s user 0.026s sys 0.045s
I20260810 22:04:58.058475 32646 raft_consensus.cc:348] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } }
I20260810 22:04:58.059862 32646 raft_consensus.cc:729] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee255d32da241aeb929742ee6005b22, State: Initialized, Role: FOLLOWER
I20260810 22:04:58.059571 32641 raft_consensus.cc:348] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:58.060523 32641 raft_consensus.cc:729] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee255d32da241aeb929742ee6005b22, State: Initialized, Role: FOLLOWER
I20260810 22:04:58.061482 32641 consensus_queue.cc:260] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:58.062249 32646 consensus_queue.cc:260] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dee255d32da241aeb929742ee6005b22" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 34103 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } }
I20260810 22:04:58.071583 32641 ts_tablet_manager.cc:1428] T f5b7bfea936e4001887b340805850f7a P dee255d32da241aeb929742ee6005b22: Time spent starting tablet: real 0.053s user 0.025s sys 0.013s
I20260810 22:04:58.072544 32646 ts_tablet_manager.cc:1428] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22: Time spent starting tablet: real 0.050s user 0.016s sys 0.006s
I20260810 22:04:58.164934 32017 raft_consensus.cc:1261] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61 [term 2 FOLLOWER]: Refusing update from remote peer e07d7c5a69cf4e2e938b11c692e3565c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260810 22:04:58.166256 32576 consensus_queue.cc:1035] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 22:04:58.183149 32110 consensus_peers.cc:487] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c -> Peer dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103): Couldn't send request to peer dee255d32da241aeb929742ee6005b22. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:34103: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:58.185180 32568 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.129:34103
I20260810 22:04:58.185403 329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.129:34103 every 8 connection(s)
I20260810 22:04:58.191792 32568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
I20260810 22:04:58.195771 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 32568
W20260810 22:04:58.204668 331 tablet.cc:2369] T 26ac18eeb4a54ec2865d5bed0b18c84a P dee255d32da241aeb929742ee6005b22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:04:58.228411 330 heartbeater.cc:347] Connected to a master server at 127.27.17.190:44707
I20260810 22:04:58.228849 330 heartbeater.cc:464] Registering TS with master...
I20260810 22:04:58.229841 330 heartbeater.cc:511] Master 127.27.17.190:44707 requested a full tablet report, sending...
I20260810 22:04:58.233847 31606 ts_manager.cc:194] Re-registered known tserver with Master: dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103)
W20260810 22:04:58.235954 31606 catalog_manager.cc:5408] Tablet 79023ddd04404bc0ba3d3e673cd80c08 (table test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]) has failed on TS dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 79023ddd04404bc0ba3d3e673cd80c08
W20260810 22:04:58.236370 31606 catalog_manager.cc:5408] Tablet f4bc97c76ce24181bbac8c1b6c0da80e (table test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]) has failed on TS dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet f4bc97c76ce24181bbac8c1b6c0da80e
I20260810 22:04:58.241951 31606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.129:37413
I20260810 22:04:58.266381 31926 consensus_queue.cc:579] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [LEADER]: The tablet replica hosted on peer dee255d32da241aeb929742ee6005b22 has failed
I20260810 22:04:58.276944 32017 consensus_queue.cc:237] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } }
I20260810 22:04:58.306463 32381 raft_consensus.cc:1261] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9 [term 1 FOLLOWER]: Refusing update from remote peer 44c1b033bb9541a7bc66207ff2deba61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:04:58.308266 341 consensus_queue.cc:1035] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 22:04:58.326006 341 raft_consensus.cc:2932] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER dee255d32da241aeb929742ee6005b22 (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } }
I20260810 22:04:58.331048 32380 raft_consensus.cc:2932] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER dee255d32da241aeb929742ee6005b22 (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } }
I20260810 22:04:58.352037 31606 catalog_manager.cc:5495] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9 reported cstate change: config changed from index -1 to 2, VOTER dee255d32da241aeb929742ee6005b22 (127.27.17.129) evicted. New cstate: current_term: 1 leader_uuid: "44c1b033bb9541a7bc66207ff2deba61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } }
I20260810 22:04:58.366344 31582 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f4bc97c76ce24181bbac8c1b6c0da80e with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 22:04:58.381716 32017 consensus_queue.cc:237] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: NON_VOTER last_known_addr { host: "127.27.17.131" port: 38509 } attrs { promote: true } }
W20260810 22:04:58.388115 31928 consensus_peers.cc:487] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 -> Peer e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131:38509): Couldn't send request to peer e07d7c5a69cf4e2e938b11c692e3565c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4bc97c76ce24181bbac8c1b6c0da80e. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:58.388317 32380 raft_consensus.cc:1261] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9 [term 1 FOLLOWER]: Refusing update from remote peer 44c1b033bb9541a7bc66207ff2deba61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 22:04:58.390092 351 consensus_queue.cc:1035] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 22:04:58.394279 32706 tablet_service.cc:1514] Processing DeleteTablet for tablet f4bc97c76ce24181bbac8c1b6c0da80e with delete_type TABLET_DATA_TOMBSTONED (TS dee255d32da241aeb929742ee6005b22 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:57952
I20260810 22:04:58.396152 341 raft_consensus.cc:2932] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: NON_VOTER last_known_addr { host: "127.27.17.131" port: 38509 } attrs { promote: true } } }
I20260810 22:04:58.397996 32380 raft_consensus.cc:2932] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: NON_VOTER last_known_addr { host: "127.27.17.131" port: 38509 } attrs { promote: true } } }
I20260810 22:04:58.408057 31582 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f4bc97c76ce24181bbac8c1b6c0da80e with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260810 22:04:58.410250 356 ts_tablet_manager.cc:1905] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 22:04:58.410372 31605 catalog_manager.cc:5495] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9 reported cstate change: config changed from index 2 to 3, NON_VOTER e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131) added. New cstate: current_term: 1 leader_uuid: "44c1b033bb9541a7bc66207ff2deba61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: NON_VOTER last_known_addr { host: "127.27.17.131" port: 38509 } attrs { promote: true } } }
I20260810 22:04:58.428915 356 ts_tablet_manager.cc:1918] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260810 22:04:58.429322 356 log.cc:1196] T f4bc97c76ce24181bbac8c1b6c0da80e P dee255d32da241aeb929742ee6005b22: Deleting WAL directory at /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/wal/wals/f4bc97c76ce24181bbac8c1b6c0da80e
I20260810 22:04:58.431051 31582 catalog_manager.cc:4841] TS dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103): tablet f4bc97c76ce24181bbac8c1b6c0da80e (table test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]) successfully deleted
I20260810 22:04:58.451187 330 heartbeater.cc:503] Master 127.27.17.190:44707 was elected leader, sending a full tablet report...
W20260810 22:04:58.464813 31606 catalog_manager.cc:5408] Tablet 79023ddd04404bc0ba3d3e673cd80c08 (table test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]) has failed on TS dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 79023ddd04404bc0ba3d3e673cd80c08
W20260810 22:04:58.465236 31606 catalog_manager.cc:5408] Tablet f4bc97c76ce24181bbac8c1b6c0da80e (table test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]) has failed on TS dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet f4bc97c76ce24181bbac8c1b6c0da80e
I20260810 22:04:58.639786 32355 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 22:04:58.640602 31985 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 22:04:58.642453 32706 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 22:04:58.665971 32170 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 22:04:58.799239 32110 consensus_queue.cc:579] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [LEADER]: The tablet replica hosted on peer dee255d32da241aeb929742ee6005b22 has failed
I20260810 22:04:58.821658 32195 consensus_queue.cc:237] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } }
I20260810 22:04:58.827602 32017 raft_consensus.cc:1261] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61 [term 2 FOLLOWER]: Refusing update from remote peer e07d7c5a69cf4e2e938b11c692e3565c: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 22:04:58.829653 387 consensus_queue.cc:1035] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 22:04:58.836362 387 raft_consensus.cc:2932] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER dee255d32da241aeb929742ee6005b22 (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } }
I20260810 22:04:58.844218 32017 raft_consensus.cc:2932] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER dee255d32da241aeb929742ee6005b22 (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } }
I20260810 22:04:58.850867 31584 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 79023ddd04404bc0ba3d3e673cd80c08 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 22:04:58.854547 31606 catalog_manager.cc:5495] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c reported cstate change: config changed from index -1 to 2, VOTER dee255d32da241aeb929742ee6005b22 (127.27.17.129) evicted. New cstate: current_term: 2 leader_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } health_report { overall_health: HEALTHY } } }
I20260810 22:04:58.870913 32706 tablet_service.cc:1514] Processing DeleteTablet for tablet 79023ddd04404bc0ba3d3e673cd80c08 with delete_type TABLET_DATA_TOMBSTONED (TS dee255d32da241aeb929742ee6005b22 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:57952
I20260810 22:04:58.871881 356 ts_tablet_manager.cc:1905] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 22:04:58.876308 32195 consensus_queue.cc:237] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: NON_VOTER last_known_addr { host: "127.27.17.132" port: 34125 } attrs { promote: true } }
I20260810 22:04:58.889786 32017 raft_consensus.cc:1261] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61 [term 2 FOLLOWER]: Refusing update from remote peer e07d7c5a69cf4e2e938b11c692e3565c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260810 22:04:58.891299 387 consensus_queue.cc:1035] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 22:04:58.897866 324 raft_consensus.cc:2932] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, NON_VOTER dd8ff3433b064de08a86dce7ab10aab9 (127.27.17.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: NON_VOTER last_known_addr { host: "127.27.17.132" port: 34125 } attrs { promote: true } } }
W20260810 22:04:58.901081 32113 consensus_peers.cc:487] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c -> Peer dd8ff3433b064de08a86dce7ab10aab9 (127.27.17.132:34125): Couldn't send request to peer dd8ff3433b064de08a86dce7ab10aab9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79023ddd04404bc0ba3d3e673cd80c08. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:04:58.900409 32017 raft_consensus.cc:2932] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, NON_VOTER dd8ff3433b064de08a86dce7ab10aab9 (127.27.17.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: NON_VOTER last_known_addr { host: "127.27.17.132" port: 34125 } attrs { promote: true } } }
I20260810 22:04:58.914009 31584 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 79023ddd04404bc0ba3d3e673cd80c08 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260810 22:04:58.917549 31606 catalog_manager.cc:5495] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c reported cstate change: config changed from index 2 to 3, NON_VOTER dd8ff3433b064de08a86dce7ab10aab9 (127.27.17.132) added. New cstate: current_term: 2 leader_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: NON_VOTER last_known_addr { host: "127.27.17.132" port: 34125 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260810 22:04:58.922699 356 ts_tablet_manager.cc:1918] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260810 22:04:58.923091 356 log.cc:1196] T 79023ddd04404bc0ba3d3e673cd80c08 P dee255d32da241aeb929742ee6005b22: Deleting WAL directory at /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/wal/wals/79023ddd04404bc0ba3d3e673cd80c08
I20260810 22:04:58.925801 31582 catalog_manager.cc:4841] TS dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103): tablet 79023ddd04404bc0ba3d3e673cd80c08 (table test-workload [id=ea7ca8a9a2434b6eb1b584c08a9633d2]) successfully deleted
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
14ffa66e458544e5900503af3474f918 | 127.27.17.190:44707 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
I20260810 22:04:58.975775 388 ts_tablet_manager.cc:927] T f4bc97c76ce24181bbac8c1b6c0da80e P e07d7c5a69cf4e2e938b11c692e3565c: Initiating tablet copy from peer 44c1b033bb9541a7bc66207ff2deba61 (127.27.17.130:46865)
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:38205 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
44c1b033bb9541a7bc66207ff2deba61 | 127.27.17.130:46865 | HEALTHY | <none> | 2 | 0
dd8ff3433b064de08a86dce7ab10aab9 | 127.27.17.132:34125 | HEALTHY | <none> | 1 | 0
dee255d32da241aeb929742ee6005b22 | 127.27.17.129:34103 | HEALTHY | <none> | 0 | 0
e07d7c5a69cf4e2e938b11c692e3565c | 127.27.17.131:38509 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
I20260810 22:04:58.977718 388 tablet_copy_client.cc:316] T f4bc97c76ce24181bbac8c1b6c0da80e P e07d7c5a69cf4e2e938b11c692e3565c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.17.130:46865
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.27.17.129:34103
env_inject_eio | 1 | hidden | 127.27.17.129:34103
env_inject_eio_globs | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-1/** | hidden | 127.27.17.129:34103
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.27.17.129 | experimental | 127.27.17.129:34103
local_ip_for_outbound_sockets | 127.27.17.130 | experimental | 127.27.17.130:46865
local_ip_for_outbound_sockets | 127.27.17.131 | experimental | 127.27.17.131:38509
local_ip_for_outbound_sockets | 127.27.17.132 | experimental | 127.27.17.132:34125
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.27.17.129:34103
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.27.17.130:46865
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.27.17.131:38509
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.27.17.132:34125
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:38205 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 26ac18eeb4a54ec2865d5bed0b18c84a of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
I20260810 22:04:58.979630 32040 tablet_copy_service.cc:140] P 44c1b033bb9541a7bc66207ff2deba61: Received BeginTabletCopySession request for tablet f4bc97c76ce24181bbac8c1b6c0da80e from peer e07d7c5a69cf4e2e938b11c692e3565c ({username='slave'} at 127.27.17.131:57485)
dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103): RUNNING
e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131:38509): RUNNING [LEADER]
dd8ff3433b064de08a86dce7ab10aab9 (127.27.17.132:34125): RUNNING
I20260810 22:04:58.980041 32040 tablet_copy_service.cc:161] P 44c1b033bb9541a7bc66207ff2deba61: Beginning new tablet copy session on tablet f4bc97c76ce24181bbac8c1b6c0da80e from peer e07d7c5a69cf4e2e938b11c692e3565c at {username='slave'} at 127.27.17.131:57485: session id = e07d7c5a69cf4e2e938b11c692e3565c-f4bc97c76ce24181bbac8c1b6c0da80e
All reported replicas are:
A = dee255d32da241aeb929742ee6005b22
B = e07d7c5a69cf4e2e938b11c692e3565c
C = dd8ff3433b064de08a86dce7ab10aab9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
Tablet 79023ddd04404bc0ba3d3e673cd80c08 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 79023ddd04404bc0ba3d3e673cd80c08
44c1b033bb9541a7bc66207ff2deba61 (127.27.17.130:46865): RUNNING
e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131:38509): RUNNING [LEADER]
All reported replicas are:
A = dee255d32da241aeb929742ee6005b22
B = 44c1b033bb9541a7bc66207ff2deba61
C = e07d7c5a69cf4e2e938b11c692e3565c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | -1 | Yes
C | A B C* | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 5 | 3 | 0 | 1 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 4
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 22:04:58.985684 32040 tablet_copy_source_session.cc:205] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:04:58.992223 27718 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 22:04:59.000684 388 tablet_copy_client.cc:791] T f4bc97c76ce24181bbac8c1b6c0da80e P e07d7c5a69cf4e2e938b11c692e3565c: tablet copy: Starting download of 0 data blocks...
I20260810 22:04:59.001243 388 tablet_copy_client.cc:655] T f4bc97c76ce24181bbac8c1b6c0da80e P e07d7c5a69cf4e2e938b11c692e3565c: tablet copy: Starting download of 1 WAL segments...
I20260810 22:04:59.004808 388 tablet_copy_client.cc:531] T f4bc97c76ce24181bbac8c1b6c0da80e P e07d7c5a69cf4e2e938b11c692e3565c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:04:59.010555 388 tablet_bootstrap.cc:492] T f4bc97c76ce24181bbac8c1b6c0da80e P e07d7c5a69cf4e2e938b11c692e3565c: Bootstrap starting.
I20260810 22:04:59.028332 388 tablet_bootstrap.cc:492] T f4bc97c76ce24181bbac8c1b6c0da80e P e07d7c5a69cf4e2e938b11c692e3565c: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:04:59.028995 388 tablet_bootstrap.cc:492] T f4bc97c76ce24181bbac8c1b6c0da80e P e07d7c5a69cf4e2e938b11c692e3565c: Bootstrap complete.
I20260810 22:04:59.029551 388 ts_tablet_manager.cc:1397] T f4bc97c76ce24181bbac8c1b6c0da80e P e07d7c5a69cf4e2e938b11c692e3565c: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.005s
I20260810 22:04:59.032153 388 raft_consensus.cc:348] T f4bc97c76ce24181bbac8c1b6c0da80e P e07d7c5a69cf4e2e938b11c692e3565c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: NON_VOTER last_known_addr { host: "127.27.17.131" port: 38509 } attrs { promote: true } }
I20260810 22:04:59.032682 388 raft_consensus.cc:729] T f4bc97c76ce24181bbac8c1b6c0da80e P e07d7c5a69cf4e2e938b11c692e3565c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e07d7c5a69cf4e2e938b11c692e3565c, State: Initialized, Role: LEARNER
I20260810 22:04:59.033089 388 consensus_queue.cc:260] T f4bc97c76ce24181bbac8c1b6c0da80e P e07d7c5a69cf4e2e938b11c692e3565c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: NON_VOTER last_known_addr { host: "127.27.17.131" port: 38509 } attrs { promote: true } }
I20260810 22:04:59.034826 388 ts_tablet_manager.cc:1428] T f4bc97c76ce24181bbac8c1b6c0da80e P e07d7c5a69cf4e2e938b11c692e3565c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:04:59.037976 32040 tablet_copy_service.cc:342] P 44c1b033bb9541a7bc66207ff2deba61: Request end of tablet copy session e07d7c5a69cf4e2e938b11c692e3565c-f4bc97c76ce24181bbac8c1b6c0da80e received from {username='slave'} at 127.27.17.131:57485
I20260810 22:04:59.038396 32040 tablet_copy_service.cc:433] P 44c1b033bb9541a7bc66207ff2deba61: ending tablet copy session e07d7c5a69cf4e2e938b11c692e3565c-f4bc97c76ce24181bbac8c1b6c0da80e on tablet f4bc97c76ce24181bbac8c1b6c0da80e with peer e07d7c5a69cf4e2e938b11c692e3565c
E20260810 22:04:59.242251 321 env_posix.cc:432] I/O error, context: INJECTED FAILURE
I20260810 22:04:59.411671 32195 raft_consensus.cc:1203] T f4bc97c76ce24181bbac8c1b6c0da80e P e07d7c5a69cf4e2e938b11c692e3565c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20260810 22:04:59.420475 421 ts_tablet_manager.cc:927] T 79023ddd04404bc0ba3d3e673cd80c08 P dd8ff3433b064de08a86dce7ab10aab9: Initiating tablet copy from peer e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131:38509)
I20260810 22:04:59.422577 421 tablet_copy_client.cc:316] T 79023ddd04404bc0ba3d3e673cd80c08 P dd8ff3433b064de08a86dce7ab10aab9: tablet copy: Beginning tablet copy session from remote peer at address 127.27.17.131:38509
I20260810 22:04:59.424630 32221 tablet_copy_service.cc:140] P e07d7c5a69cf4e2e938b11c692e3565c: Received BeginTabletCopySession request for tablet 79023ddd04404bc0ba3d3e673cd80c08 from peer dd8ff3433b064de08a86dce7ab10aab9 ({username='slave'} at 127.27.17.132:34539)
I20260810 22:04:59.425179 32221 tablet_copy_service.cc:161] P e07d7c5a69cf4e2e938b11c692e3565c: Beginning new tablet copy session on tablet 79023ddd04404bc0ba3d3e673cd80c08 from peer dd8ff3433b064de08a86dce7ab10aab9 at {username='slave'} at 127.27.17.132:34539: session id = dd8ff3433b064de08a86dce7ab10aab9-79023ddd04404bc0ba3d3e673cd80c08
I20260810 22:04:59.431968 32221 tablet_copy_source_session.cc:205] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:04:59.452373 421 tablet_copy_client.cc:791] T 79023ddd04404bc0ba3d3e673cd80c08 P dd8ff3433b064de08a86dce7ab10aab9: tablet copy: Starting download of 0 data blocks...
I20260810 22:04:59.453171 421 tablet_copy_client.cc:655] T 79023ddd04404bc0ba3d3e673cd80c08 P dd8ff3433b064de08a86dce7ab10aab9: tablet copy: Starting download of 1 WAL segments...
I20260810 22:04:59.472563 421 tablet_copy_client.cc:531] T 79023ddd04404bc0ba3d3e673cd80c08 P dd8ff3433b064de08a86dce7ab10aab9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:04:59.482308 421 tablet_bootstrap.cc:492] T 79023ddd04404bc0ba3d3e673cd80c08 P dd8ff3433b064de08a86dce7ab10aab9: Bootstrap starting.
I20260810 22:04:59.517549 31985 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 22:04:59.518123 421 tablet_bootstrap.cc:492] T 79023ddd04404bc0ba3d3e673cd80c08 P dd8ff3433b064de08a86dce7ab10aab9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:04:59.519061 421 tablet_bootstrap.cc:492] T 79023ddd04404bc0ba3d3e673cd80c08 P dd8ff3433b064de08a86dce7ab10aab9: Bootstrap complete.
I20260810 22:04:59.519070 32355 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 22:04:59.519801 421 ts_tablet_manager.cc:1397] T 79023ddd04404bc0ba3d3e673cd80c08 P dd8ff3433b064de08a86dce7ab10aab9: Time spent bootstrapping tablet: real 0.038s user 0.026s sys 0.004s
I20260810 22:04:59.522089 32170 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 22:04:59.522507 421 raft_consensus.cc:348] T 79023ddd04404bc0ba3d3e673cd80c08 P dd8ff3433b064de08a86dce7ab10aab9 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: NON_VOTER last_known_addr { host: "127.27.17.132" port: 34125 } attrs { promote: true } }
I20260810 22:04:59.523257 421 raft_consensus.cc:729] T 79023ddd04404bc0ba3d3e673cd80c08 P dd8ff3433b064de08a86dce7ab10aab9 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: dd8ff3433b064de08a86dce7ab10aab9, State: Initialized, Role: LEARNER
I20260810 22:04:59.523866 421 consensus_queue.cc:260] T 79023ddd04404bc0ba3d3e673cd80c08 P dd8ff3433b064de08a86dce7ab10aab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: NON_VOTER last_known_addr { host: "127.27.17.132" port: 34125 } attrs { promote: true } }
I20260810 22:04:59.524818 32706 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 22:04:59.526049 421 ts_tablet_manager.cc:1428] T 79023ddd04404bc0ba3d3e673cd80c08 P dd8ff3433b064de08a86dce7ab10aab9: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260810 22:04:59.528345 32221 tablet_copy_service.cc:342] P e07d7c5a69cf4e2e938b11c692e3565c: Request end of tablet copy session dd8ff3433b064de08a86dce7ab10aab9-79023ddd04404bc0ba3d3e673cd80c08 received from {username='slave'} at 127.27.17.132:34539
I20260810 22:04:59.528728 32221 tablet_copy_service.cc:433] P e07d7c5a69cf4e2e938b11c692e3565c: ending tablet copy session dd8ff3433b064de08a86dce7ab10aab9-79023ddd04404bc0ba3d3e673cd80c08 on tablet 79023ddd04404bc0ba3d3e673cd80c08 with peer dd8ff3433b064de08a86dce7ab10aab9
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
14ffa66e458544e5900503af3474f918 | 127.27.17.190:44707 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:38205 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
44c1b033bb9541a7bc66207ff2deba61 | 127.27.17.130:46865 | HEALTHY | <none> | 2 | 0
dd8ff3433b064de08a86dce7ab10aab9 | 127.27.17.132:34125 | HEALTHY | <none> | 1 | 0
dee255d32da241aeb929742ee6005b22 | 127.27.17.129:34103 | HEALTHY | <none> | 0 | 0
e07d7c5a69cf4e2e938b11c692e3565c | 127.27.17.131:38509 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.27.17.129:34103
env_inject_eio | 1 | hidden | 127.27.17.129:34103
env_inject_eio_globs | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-1/** | hidden | 127.27.17.129:34103
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.27.17.129 | experimental | 127.27.17.129:34103
local_ip_for_outbound_sockets | 127.27.17.130 | experimental | 127.27.17.130:46865
local_ip_for_outbound_sockets | 127.27.17.131 | experimental | 127.27.17.131:38509
local_ip_for_outbound_sockets | 127.27.17.132 | experimental | 127.27.17.132:34125
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.27.17.129:34103
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.27.17.130:46865
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.27.17.131:38509
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.27.17.132:34125
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:38205 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 79023ddd04404bc0ba3d3e673cd80c08 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
44c1b033bb9541a7bc66207ff2deba61 (127.27.17.130:46865): RUNNING
e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131:38509): RUNNING [LEADER]
dd8ff3433b064de08a86dce7ab10aab9 (127.27.17.132:34125): not running [NONVOTER]
State: BOOTSTRAPPING
Data state: TABLET_DATA_READY
Last status: Bootstrap starting.
All reported replicas are:
A = 44c1b033bb9541a7bc66207ff2deba61
B = e07d7c5a69cf4e2e938b11c692e3565c
C = dd8ff3433b064de08a86dce7ab10aab9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* C~ | 2 | 3 | Yes
B | A B* C~ | 2 | 3 | Yes
C | A B C~ | 2 | 3 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 5 | 4 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 4
Median | 4
Third Quartile | 5
Maximum | 5
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | dee255d32da241aeb929742ee6005b22 | 127.27.17.129:34103 | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 22:04:59.794644 27718 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 22:04:59.819666 32380 raft_consensus.cc:1203] T 79023ddd04404bc0ba3d3e673cd80c08 P dd8ff3433b064de08a86dce7ab10aab9 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260810 22:04:59.847983 396 raft_consensus.cc:1050] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61: attempting to promote NON_VOTER e07d7c5a69cf4e2e938b11c692e3565c to VOTER
I20260810 22:04:59.850314 396 consensus_queue.cc:237] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } attrs { promote: false } }
I20260810 22:04:59.855593 32195 raft_consensus.cc:1261] T f4bc97c76ce24181bbac8c1b6c0da80e P e07d7c5a69cf4e2e938b11c692e3565c [term 1 LEARNER]: Refusing update from remote peer 44c1b033bb9541a7bc66207ff2deba61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 22:04:59.856026 32380 raft_consensus.cc:1261] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9 [term 1 FOLLOWER]: Refusing update from remote peer 44c1b033bb9541a7bc66207ff2deba61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 22:04:59.857074 392 consensus_queue.cc:1035] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 22:04:59.858057 341 consensus_queue.cc:1035] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 22:04:59.864907 392 raft_consensus.cc:2932] T f4bc97c76ce24181bbac8c1b6c0da80e P 44c1b033bb9541a7bc66207ff2deba61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } attrs { promote: false } } }
I20260810 22:04:59.866719 32380 raft_consensus.cc:2932] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } attrs { promote: false } } }
I20260810 22:04:59.866412 32195 raft_consensus.cc:2932] T f4bc97c76ce24181bbac8c1b6c0da80e P e07d7c5a69cf4e2e938b11c692e3565c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } attrs { promote: false } } }
I20260810 22:04:59.878254 31605 catalog_manager.cc:5495] T f4bc97c76ce24181bbac8c1b6c0da80e P dd8ff3433b064de08a86dce7ab10aab9 reported cstate change: config changed from index 3 to 4, e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "44c1b033bb9541a7bc66207ff2deba61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } } peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } attrs { promote: false } } }
I20260810 22:05:00.273222 444 raft_consensus.cc:1050] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c: attempting to promote NON_VOTER dd8ff3433b064de08a86dce7ab10aab9 to VOTER
I20260810 22:05:00.275350 444 consensus_queue.cc:237] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } attrs { promote: false } }
I20260810 22:05:00.301069 32017 raft_consensus.cc:1261] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61 [term 2 FOLLOWER]: Refusing update from remote peer e07d7c5a69cf4e2e938b11c692e3565c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260810 22:05:00.302489 387 consensus_queue.cc:1035] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 22:05:00.307320 32380 raft_consensus.cc:1261] T 79023ddd04404bc0ba3d3e673cd80c08 P dd8ff3433b064de08a86dce7ab10aab9 [term 2 LEARNER]: Refusing update from remote peer e07d7c5a69cf4e2e938b11c692e3565c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260810 22:05:00.308857 387 consensus_queue.cc:1035] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 22:05:00.332660 31985 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 22:05:00.343243 444 raft_consensus.cc:2932] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, dd8ff3433b064de08a86dce7ab10aab9 (127.27.17.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } attrs { promote: false } } }
I20260810 22:05:00.350883 32355 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 22:05:00.356565 32380 raft_consensus.cc:2932] T 79023ddd04404bc0ba3d3e673cd80c08 P dd8ff3433b064de08a86dce7ab10aab9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, dd8ff3433b064de08a86dce7ab10aab9 (127.27.17.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } attrs { promote: false } } }
I20260810 22:05:00.363082 32017 raft_consensus.cc:2932] T 79023ddd04404bc0ba3d3e673cd80c08 P 44c1b033bb9541a7bc66207ff2deba61 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, dd8ff3433b064de08a86dce7ab10aab9 (127.27.17.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } attrs { promote: false } } }
I20260810 22:05:00.377701 31606 catalog_manager.cc:5495] T 79023ddd04404bc0ba3d3e673cd80c08 P e07d7c5a69cf4e2e938b11c692e3565c reported cstate change: config changed from index 3 to 4, dd8ff3433b064de08a86dce7ab10aab9 (127.27.17.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44c1b033bb9541a7bc66207ff2deba61" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 46865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07d7c5a69cf4e2e938b11c692e3565c" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 38509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd8ff3433b064de08a86dce7ab10aab9" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 34125 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 22:05:00.387051 32170 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 22:05:00.404089 32706 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
14ffa66e458544e5900503af3474f918 | 127.27.17.190:44707 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:38205 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
44c1b033bb9541a7bc66207ff2deba61 | 127.27.17.130:46865 | HEALTHY | <none> | 2 | 0
dd8ff3433b064de08a86dce7ab10aab9 | 127.27.17.132:34125 | HEALTHY | <none> | 1 | 0
dee255d32da241aeb929742ee6005b22 | 127.27.17.129:34103 | HEALTHY | <none> | 0 | 0
e07d7c5a69cf4e2e938b11c692e3565c | 127.27.17.131:38509 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.27.17.129:34103
env_inject_eio | 1 | hidden | 127.27.17.129:34103
env_inject_eio_globs | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-1/** | hidden | 127.27.17.129:34103
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.27.17.129 | experimental | 127.27.17.129:34103
local_ip_for_outbound_sockets | 127.27.17.130 | experimental | 127.27.17.130:46865
local_ip_for_outbound_sockets | 127.27.17.131 | experimental | 127.27.17.131:38509
local_ip_for_outbound_sockets | 127.27.17.132 | experimental | 127.27.17.132:34125
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.27.17.129:34103
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.27.17.130:46865
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.27.17.131:38509
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.27.17.132:34125
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:38205 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 79023ddd04404bc0ba3d3e673cd80c08 of table 'test-workload' is conflicted: 3 replicas' active configs disagree with the leader master's
44c1b033bb9541a7bc66207ff2deba61 (127.27.17.130:46865): RUNNING
e07d7c5a69cf4e2e938b11c692e3565c (127.27.17.131:38509): RUNNING [LEADER]
dd8ff3433b064de08a86dce7ab10aab9 (127.27.17.132:34125): RUNNING [NONVOTER]
All reported replicas are:
A = 44c1b033bb9541a7bc66207ff2deba61
B = e07d7c5a69cf4e2e938b11c692e3565c
C = dd8ff3433b064de08a86dce7ab10aab9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* C | 2 | 4 | No
B | A B* C | 2 | 4 | Yes
C | A B* C | 2 | 4 | No
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 5 | 4 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 4
Median | 4
Third Quartile | 5
Maximum | 5
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | dee255d32da241aeb929742ee6005b22 | 127.27.17.129:34103 | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 22:05:00.650900 27718 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 22:05:01.292932 32355 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 22:05:01.298890 31985 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260810 22:05:01.324784 32706 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 22:05:01.339915 32170 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
14ffa66e458544e5900503af3474f918 | 127.27.17.190:44707 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:38205 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
44c1b033bb9541a7bc66207ff2deba61 | 127.27.17.130:46865 | HEALTHY | <none> | 2 | 0
dd8ff3433b064de08a86dce7ab10aab9 | 127.27.17.132:34125 | HEALTHY | <none> | 1 | 0
dee255d32da241aeb929742ee6005b22 | 127.27.17.129:34103 | HEALTHY | <none> | 0 | 0
e07d7c5a69cf4e2e938b11c692e3565c | 127.27.17.131:38509 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.27.17.129:34103
env_inject_eio | 1 | hidden | 127.27.17.129:34103
env_inject_eio_globs | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-1/** | hidden | 127.27.17.129:34103
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.27.17.129 | experimental | 127.27.17.129:34103
local_ip_for_outbound_sockets | 127.27.17.130 | experimental | 127.27.17.130:46865
local_ip_for_outbound_sockets | 127.27.17.131 | experimental | 127.27.17.131:38509
local_ip_for_outbound_sockets | 127.27.17.132 | experimental | 127.27.17.132:34125
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.27.17.129:34103
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.27.17.130:46865
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.27.17.131:38509
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786399438989427-27718-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.27.17.132:34125
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:38205 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 5 | 5 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 4
Median | 4
Third Quartile | 5
Maximum | 5
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | dee255d32da241aeb929742ee6005b22 | 127.27.17.129:34103 | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260810 22:05:01.610378 27718 log_verifier.cc:126] Checking tablet 0bac2f72b88b4c74a8ef33e484828638
I20260810 22:05:01.675359 27718 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 0bac2f72b88b4c74a8ef33e484828638
I20260810 22:05:01.675577 27718 log_verifier.cc:126] Checking tablet 26ac18eeb4a54ec2865d5bed0b18c84a
I20260810 22:05:01.740211 27718 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 26ac18eeb4a54ec2865d5bed0b18c84a
I20260810 22:05:01.740415 27718 log_verifier.cc:126] Checking tablet 79023ddd04404bc0ba3d3e673cd80c08
I20260810 22:05:01.803810 27718 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 79023ddd04404bc0ba3d3e673cd80c08
I20260810 22:05:01.804034 27718 log_verifier.cc:126] Checking tablet f4bc97c76ce24181bbac8c1b6c0da80e
I20260810 22:05:01.867681 27718 log_verifier.cc:177] Verified matching terms for 4 ops in tablet f4bc97c76ce24181bbac8c1b6c0da80e
I20260810 22:05:01.867890 27718 log_verifier.cc:126] Checking tablet f5b7bfea936e4001887b340805850f7a
I20260810 22:05:01.926608 27718 log_verifier.cc:177] Verified matching terms for 1 ops in tablet f5b7bfea936e4001887b340805850f7a
I20260810 22:05:02.360885 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 32568
W20260810 22:05:02.429132 32110 connection.cc:535] client connection to 127.27.17.129:34103 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 22:05:02.430276 32110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 7 similar messages]
I20260810 22:05:02.440167 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 31893
W20260810 22:05:02.461433 32110 consensus_peers.cc:487] T 26ac18eeb4a54ec2865d5bed0b18c84a P e07d7c5a69cf4e2e938b11c692e3565c -> Peer dee255d32da241aeb929742ee6005b22 (127.27.17.129:34103): Couldn't send request to peer dee255d32da241aeb929742ee6005b22. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:34103: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 22:05:02.486425 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 32077
W20260810 22:05:02.495155 32291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.130:46865: connect: Connection refused (error 111)
W20260810 22:05:02.500800 32291 consensus_peers.cc:487] T 0bac2f72b88b4c74a8ef33e484828638 P dd8ff3433b064de08a86dce7ab10aab9 -> Peer 44c1b033bb9541a7bc66207ff2deba61 (127.27.17.130:46865): Couldn't send request to peer 44c1b033bb9541a7bc66207ff2deba61. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.130:46865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 22:05:02.539760 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 32260
I20260810 22:05:02.582612 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 31554
2026-08-10T22:05:02Z chronyd exiting
[ OK ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/3 (17240 ms)
[ RUN ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/4
2026-08-10T22:05:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:05:02Z Disabled control of system clock
I20260810 22:05:02.713760 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.190:37855
--webserver_interface=127.27.17.190
--webserver_port=0
--builtin_ntp_servers=127.27.17.148:37545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.190:37855
--raft_prepare_replacement_before_eviction=false with env {}
W20260810 22:05:03.031778 559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:05:03.032490 559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:05:03.033000 559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:05:03.066643 559 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260810 22:05:03.067077 559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:05:03.067349 559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:05:03.067617 559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:05:03.067867 559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:05:03.107088 559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:37545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.190:37855
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.190:37855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
--webserver_interface=127.27.17.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:05:03.108394 559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:05:03.110067 559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:05:03.128501 571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:03.128502 573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:03.129279 569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:03.130966 559 server_base.cc:1034] running on GCE node
I20260810 22:05:04.325218 559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:05:04.328709 559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:05:04.330164 559 hybrid_clock.cc:648] HybridClock initialized: now 1786399504330173 us; error 91 us; skew 500 ppm
I20260810 22:05:04.330999 559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:05:04.344522 559 webserver.cc:460] Webserver started at http://127.27.17.190:45401/ using document root <none> and password file <none>
I20260810 22:05:04.346191 559 fs_manager.cc:359] Metadata directory not provided
I20260810 22:05:04.346414 559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:05:04.347048 559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:05:04.355042 559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-0/instance:
uuid: "04703694c11e43c7b7c7470a0a35862b"
format_stamp: "Formatted at 2026-08-10 22:05:04 on dist-test-slave-bw88"
I20260810 22:05:04.356392 559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-1/instance:
uuid: "04703694c11e43c7b7c7470a0a35862b"
format_stamp: "Formatted at 2026-08-10 22:05:04 on dist-test-slave-bw88"
I20260810 22:05:04.357369 559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-2/instance:
uuid: "04703694c11e43c7b7c7470a0a35862b"
format_stamp: "Formatted at 2026-08-10 22:05:04 on dist-test-slave-bw88"
I20260810 22:05:04.358227 559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-3/instance:
uuid: "04703694c11e43c7b7c7470a0a35862b"
format_stamp: "Formatted at 2026-08-10 22:05:04 on dist-test-slave-bw88"
I20260810 22:05:04.359071 559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-4/instance:
uuid: "04703694c11e43c7b7c7470a0a35862b"
format_stamp: "Formatted at 2026-08-10 22:05:04 on dist-test-slave-bw88"
I20260810 22:05:04.359936 559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/wal/instance:
uuid: "04703694c11e43c7b7c7470a0a35862b"
format_stamp: "Formatted at 2026-08-10 22:05:04 on dist-test-slave-bw88"
I20260810 22:05:04.383895 559 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.016s sys 0.009s
I20260810 22:05:04.919652 559 fs_manager.cc:725] Time spent opening block manager: real 0.531s user 0.003s sys 0.004s
I20260810 22:05:04.920049 559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/wal
uuid: "04703694c11e43c7b7c7470a0a35862b"
format_stamp: "Formatted at 2026-08-10 22:05:04 on dist-test-slave-bw88"
I20260810 22:05:04.920476 559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:05:04.959900 559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:05:04.961503 559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:05:04.961977 559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:05:05.015810 559 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.190:37855
I20260810 22:05:05.016021 773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.190:37855 every 8 connection(s)
I20260810 22:05:05.018589 559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
I20260810 22:05:05.026937 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 559
I20260810 22:05:05.027406 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/wal/instance
I20260810 22:05:05.046108 774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b: Bootstrap starting.
I20260810 22:05:05.052840 774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:05.054828 774 log.cc:824] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b: Log is configured to *not* fsync() on all Append() calls
I20260810 22:05:05.078811 774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b: No bootstrap required, opened a new log
I20260810 22:05:05.103134 774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04703694c11e43c7b7c7470a0a35862b" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 37855 } }
I20260810 22:05:05.103837 774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:05.104107 774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04703694c11e43c7b7c7470a0a35862b, State: Initialized, Role: FOLLOWER
I20260810 22:05:05.104899 774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04703694c11e43c7b7c7470a0a35862b" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 37855 } }
I20260810 22:05:05.105448 774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:05:05.105695 774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:05:05.105947 774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:05.110406 774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04703694c11e43c7b7c7470a0a35862b" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 37855 } }
I20260810 22:05:05.111083 774 leader_election.cc:302] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04703694c11e43c7b7c7470a0a35862b; no voters:
I20260810 22:05:05.113090 774 leader_election.cc:288] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:05:05.113878 781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:05.116477 781 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b [term 1 LEADER]: Becoming Leader. State: Replica: 04703694c11e43c7b7c7470a0a35862b, State: Running, Role: LEADER
I20260810 22:05:05.117288 781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04703694c11e43c7b7c7470a0a35862b" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 37855 } }
I20260810 22:05:05.118709 774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:05:05.129635 783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04703694c11e43c7b7c7470a0a35862b. Latest consensus state: current_term: 1 leader_uuid: "04703694c11e43c7b7c7470a0a35862b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04703694c11e43c7b7c7470a0a35862b" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 37855 } } }
I20260810 22:05:05.130422 783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b [sys.catalog]: This master's current role is: LEADER
I20260810 22:05:05.129652 782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04703694c11e43c7b7c7470a0a35862b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04703694c11e43c7b7c7470a0a35862b" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 37855 } } }
I20260810 22:05:05.130892 782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b [sys.catalog]: This master's current role is: LEADER
I20260810 22:05:05.133795 793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:05:05.145649 793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:05:05.165378 793 catalog_manager.cc:1347] Generated new cluster ID: 46b78e72a33e43eba9c5c82ae96f5368
I20260810 22:05:05.165704 793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:05:05.196350 793 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:05:05.198408 793 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:05:05.212651 793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04703694c11e43c7b7c7470a0a35862b: Generated new TSK 0
I20260810 22:05:05.213961 793 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:05:05.227166 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.129:0
--local_ip_for_outbound_sockets=127.27.17.129
--webserver_interface=127.27.17.129
--webserver_port=0
--tserver_master_addrs=127.27.17.190:37855
--builtin_ntp_servers=127.27.17.148:37545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20260810 22:05:05.571444 806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:05:05.572043 806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:05:05.572611 806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:05:05.604995 806 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260810 22:05:05.605487 806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:05:05.606320 806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.129
I20260810 22:05:05.643364 806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:37545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.27.17.129
--webserver_port=0
--tserver_master_addrs=127.27.17.190:37855
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.129
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:05:05.644881 806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:05:05.646754 806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:05:05.670146 815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:05.670288 813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:05.671572 806 server_base.cc:1034] running on GCE node
W20260810 22:05:05.671407 812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:06.889166 806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:05:06.891593 806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:05:06.892958 806 hybrid_clock.cc:648] HybridClock initialized: now 1786399506892913 us; error 46 us; skew 500 ppm
I20260810 22:05:06.893818 806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:05:06.900324 806 webserver.cc:460] Webserver started at http://127.27.17.129:33191/ using document root <none> and password file <none>
I20260810 22:05:06.901858 806 fs_manager.cc:359] Metadata directory not provided
I20260810 22:05:06.902055 806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:05:06.902684 806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:05:06.907931 806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-0/instance:
uuid: "e44427371e0c485eb96e803fd122ad5f"
format_stamp: "Formatted at 2026-08-10 22:05:06 on dist-test-slave-bw88"
I20260810 22:05:06.909080 806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-1/instance:
uuid: "e44427371e0c485eb96e803fd122ad5f"
format_stamp: "Formatted at 2026-08-10 22:05:06 on dist-test-slave-bw88"
I20260810 22:05:06.910104 806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-2/instance:
uuid: "e44427371e0c485eb96e803fd122ad5f"
format_stamp: "Formatted at 2026-08-10 22:05:06 on dist-test-slave-bw88"
I20260810 22:05:06.911108 806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-3/instance:
uuid: "e44427371e0c485eb96e803fd122ad5f"
format_stamp: "Formatted at 2026-08-10 22:05:06 on dist-test-slave-bw88"
I20260810 22:05:06.912132 806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-4/instance:
uuid: "e44427371e0c485eb96e803fd122ad5f"
format_stamp: "Formatted at 2026-08-10 22:05:06 on dist-test-slave-bw88"
I20260810 22:05:06.913060 806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/wal/instance:
uuid: "e44427371e0c485eb96e803fd122ad5f"
format_stamp: "Formatted at 2026-08-10 22:05:06 on dist-test-slave-bw88"
I20260810 22:05:06.937070 806 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.014s sys 0.009s
I20260810 22:05:07.402571 806 fs_manager.cc:725] Time spent opening block manager: real 0.460s user 0.004s sys 0.000s
I20260810 22:05:07.403069 806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/wal
uuid: "e44427371e0c485eb96e803fd122ad5f"
format_stamp: "Formatted at 2026-08-10 22:05:06 on dist-test-slave-bw88"
I20260810 22:05:07.403522 806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:05:07.437975 806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:05:07.439507 806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:05:07.439980 806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:05:07.442342 806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:05:07.446246 806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:05:07.446444 806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:07.446763 806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:05:07.446918 806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:07.574343 806 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.129:32991
I20260810 22:05:07.574591 1094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.129:32991 every 8 connection(s)
I20260810 22:05:07.577066 806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
I20260810 22:05:07.584200 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 806
I20260810 22:05:07.584735 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/wal/instance
I20260810 22:05:07.591337 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.130:0
--local_ip_for_outbound_sockets=127.27.17.130
--webserver_interface=127.27.17.130
--webserver_port=0
--tserver_master_addrs=127.27.17.190:37855
--builtin_ntp_servers=127.27.17.148:37545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20260810 22:05:07.601550 1096 heartbeater.cc:347] Connected to a master server at 127.27.17.190:37855
I20260810 22:05:07.602054 1096 heartbeater.cc:464] Registering TS with master...
I20260810 22:05:07.603304 1096 heartbeater.cc:511] Master 127.27.17.190:37855 requested a full tablet report, sending...
I20260810 22:05:07.606204 721 ts_manager.cc:194] Registered new tserver with Master: e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991)
I20260810 22:05:07.608258 721 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.129:52821
W20260810 22:05:07.905326 1102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:05:07.905942 1102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:05:07.906456 1102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:05:07.937654 1102 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260810 22:05:07.938062 1102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:05:07.938856 1102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.130
I20260810 22:05:07.972741 1102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:37545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
--webserver_interface=127.27.17.130
--webserver_port=0
--tserver_master_addrs=127.27.17.190:37855
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.130
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:05:07.973974 1102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:05:07.975524 1102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:05:07.993757 1109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:08.613193 1096 heartbeater.cc:503] Master 127.27.17.190:37855 was elected leader, sending a full tablet report...
W20260810 22:05:07.994521 1115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:07.993808 1112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:09.179796 1114 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 22:05:09.179917 1102 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 22:05:09.184275 1102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:05:09.186595 1102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:05:09.187945 1102 hybrid_clock.cc:648] HybridClock initialized: now 1786399509187919 us; error 47 us; skew 500 ppm
I20260810 22:05:09.188707 1102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:05:09.195924 1102 webserver.cc:460] Webserver started at http://127.27.17.130:34393/ using document root <none> and password file <none>
I20260810 22:05:09.197474 1102 fs_manager.cc:359] Metadata directory not provided
I20260810 22:05:09.197733 1102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:05:09.198398 1102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:05:09.203544 1102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-0/instance:
uuid: "912f44f41ac346ffa75962a42491ab64"
format_stamp: "Formatted at 2026-08-10 22:05:09 on dist-test-slave-bw88"
I20260810 22:05:09.204733 1102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-1/instance:
uuid: "912f44f41ac346ffa75962a42491ab64"
format_stamp: "Formatted at 2026-08-10 22:05:09 on dist-test-slave-bw88"
I20260810 22:05:09.205735 1102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-2/instance:
uuid: "912f44f41ac346ffa75962a42491ab64"
format_stamp: "Formatted at 2026-08-10 22:05:09 on dist-test-slave-bw88"
I20260810 22:05:09.206625 1102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-3/instance:
uuid: "912f44f41ac346ffa75962a42491ab64"
format_stamp: "Formatted at 2026-08-10 22:05:09 on dist-test-slave-bw88"
I20260810 22:05:09.207502 1102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-4/instance:
uuid: "912f44f41ac346ffa75962a42491ab64"
format_stamp: "Formatted at 2026-08-10 22:05:09 on dist-test-slave-bw88"
I20260810 22:05:09.208401 1102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/wal/instance:
uuid: "912f44f41ac346ffa75962a42491ab64"
format_stamp: "Formatted at 2026-08-10 22:05:09 on dist-test-slave-bw88"
I20260810 22:05:09.232589 1102 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.017s sys 0.008s
I20260810 22:05:09.667014 1102 fs_manager.cc:725] Time spent opening block manager: real 0.429s user 0.003s sys 0.001s
I20260810 22:05:09.667395 1102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/wal
uuid: "912f44f41ac346ffa75962a42491ab64"
format_stamp: "Formatted at 2026-08-10 22:05:09 on dist-test-slave-bw88"
I20260810 22:05:09.667797 1102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:05:09.710027 1102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:05:09.711627 1102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:05:09.712136 1102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:05:09.714402 1102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:05:09.718305 1102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:05:09.718537 1102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:09.718869 1102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:05:09.719024 1102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:09.845330 1102 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.130:39307
I20260810 22:05:09.845575 1395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.130:39307 every 8 connection(s)
I20260810 22:05:09.847975 1102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
I20260810 22:05:09.855491 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 1102
I20260810 22:05:09.856346 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/wal/instance
I20260810 22:05:09.864840 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.131:0
--local_ip_for_outbound_sockets=127.27.17.131
--webserver_interface=127.27.17.131
--webserver_port=0
--tserver_master_addrs=127.27.17.190:37855
--builtin_ntp_servers=127.27.17.148:37545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20260810 22:05:09.874166 1396 heartbeater.cc:347] Connected to a master server at 127.27.17.190:37855
I20260810 22:05:09.874660 1396 heartbeater.cc:464] Registering TS with master...
I20260810 22:05:09.875630 1396 heartbeater.cc:511] Master 127.27.17.190:37855 requested a full tablet report, sending...
I20260810 22:05:09.877909 721 ts_manager.cc:194] Registered new tserver with Master: 912f44f41ac346ffa75962a42491ab64 (127.27.17.130:39307)
I20260810 22:05:09.879420 721 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.130:50085
W20260810 22:05:10.182210 1402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:05:10.182772 1402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:05:10.183287 1402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:05:10.214465 1402 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260810 22:05:10.214886 1402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:05:10.215690 1402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.131
I20260810 22:05:10.250765 1402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:37545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
--webserver_interface=127.27.17.131
--webserver_port=0
--tserver_master_addrs=127.27.17.190:37855
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.131
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:05:10.252113 1402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:05:10.253788 1402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:05:10.274022 1410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:10.884133 1396 heartbeater.cc:503] Master 127.27.17.190:37855 was elected leader, sending a full tablet report...
W20260810 22:05:10.275599 1409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:10.274047 1413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:10.276613 1402 server_base.cc:1034] running on GCE node
I20260810 22:05:11.470232 1402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:05:11.472677 1402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:05:11.474084 1402 hybrid_clock.cc:648] HybridClock initialized: now 1786399511474056 us; error 48 us; skew 500 ppm
I20260810 22:05:11.475011 1402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:05:11.481707 1402 webserver.cc:460] Webserver started at http://127.27.17.131:33491/ using document root <none> and password file <none>
I20260810 22:05:11.483155 1402 fs_manager.cc:359] Metadata directory not provided
I20260810 22:05:11.483402 1402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:05:11.484093 1402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:05:11.489971 1402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-0/instance:
uuid: "067bae0bd4a64f839e420a2f323862a5"
format_stamp: "Formatted at 2026-08-10 22:05:11 on dist-test-slave-bw88"
I20260810 22:05:11.491101 1402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-1/instance:
uuid: "067bae0bd4a64f839e420a2f323862a5"
format_stamp: "Formatted at 2026-08-10 22:05:11 on dist-test-slave-bw88"
I20260810 22:05:11.491991 1402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-2/instance:
uuid: "067bae0bd4a64f839e420a2f323862a5"
format_stamp: "Formatted at 2026-08-10 22:05:11 on dist-test-slave-bw88"
I20260810 22:05:11.492858 1402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-3/instance:
uuid: "067bae0bd4a64f839e420a2f323862a5"
format_stamp: "Formatted at 2026-08-10 22:05:11 on dist-test-slave-bw88"
I20260810 22:05:11.494024 1402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-4/instance:
uuid: "067bae0bd4a64f839e420a2f323862a5"
format_stamp: "Formatted at 2026-08-10 22:05:11 on dist-test-slave-bw88"
I20260810 22:05:11.494905 1402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/wal/instance:
uuid: "067bae0bd4a64f839e420a2f323862a5"
format_stamp: "Formatted at 2026-08-10 22:05:11 on dist-test-slave-bw88"
I20260810 22:05:11.518728 1402 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.019s sys 0.005s
I20260810 22:05:12.019464 1402 fs_manager.cc:725] Time spent opening block manager: real 0.496s user 0.004s sys 0.002s
I20260810 22:05:12.019815 1402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/wal
uuid: "067bae0bd4a64f839e420a2f323862a5"
format_stamp: "Formatted at 2026-08-10 22:05:11 on dist-test-slave-bw88"
I20260810 22:05:12.020206 1402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:05:12.057130 1402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:05:12.058602 1402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:05:12.059024 1402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:05:12.061039 1402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:05:12.064900 1402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:05:12.065106 1402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:12.065303 1402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:05:12.065507 1402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260810 22:05:12.191866 1402 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.131:41905
I20260810 22:05:12.192178 1694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.131:41905 every 8 connection(s)
I20260810 22:05:12.194494 1402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
I20260810 22:05:12.195509 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 1402
I20260810 22:05:12.196107 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/wal/instance
I20260810 22:05:12.220679 1695 heartbeater.cc:347] Connected to a master server at 127.27.17.190:37855
I20260810 22:05:12.221123 1695 heartbeater.cc:464] Registering TS with master...
I20260810 22:05:12.222098 1695 heartbeater.cc:511] Master 127.27.17.190:37855 requested a full tablet report, sending...
I20260810 22:05:12.224298 721 ts_manager.cc:194] Registered new tserver with Master: 067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905)
I20260810 22:05:12.225831 721 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.131:34397
I20260810 22:05:12.232650 27718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:05:12.266641 27718 test_util.cc:268] Using random seed: -1940191364
I20260810 22:05:12.311527 721 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52238:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\231\231\231\031\004\001\0002333\004\001\000\313\314\314L\004\001\000dfff"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 22:05:12.396864 1304 tablet_service.cc:1467] Processing CreateTablet for tablet 33a9afb6e00543c184b1ef3bb975cc20 (DEFAULT_TABLE table=test-workload [id=122e1e087f99407c8276bb8c72ac7970]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260810 22:05:12.399824 1303 tablet_service.cc:1467] Processing CreateTablet for tablet 92495f335e074d6fa01e66d395101407 (DEFAULT_TABLE table=test-workload [id=122e1e087f99407c8276bb8c72ac7970]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260810 22:05:12.400637 1001 tablet_service.cc:1467] Processing CreateTablet for tablet 2e33aa436d984401993daed75b7a41b0 (DEFAULT_TABLE table=test-workload [id=122e1e087f99407c8276bb8c72ac7970]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260810 22:05:12.396863 1307 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0192a67e9e45a78a45868b1ce6da9d (DEFAULT_TABLE table=test-workload [id=122e1e087f99407c8276bb8c72ac7970]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260810 22:05:12.401075 999 tablet_service.cc:1467] Processing CreateTablet for tablet 632fa1d756704125b84b5e1772cb9076 (DEFAULT_TABLE table=test-workload [id=122e1e087f99407c8276bb8c72ac7970]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260810 22:05:12.397360 1306 tablet_service.cc:1467] Processing CreateTablet for tablet 632fa1d756704125b84b5e1772cb9076 (DEFAULT_TABLE table=test-workload [id=122e1e087f99407c8276bb8c72ac7970]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260810 22:05:12.402441 1310 tablet_service.cc:1467] Processing CreateTablet for tablet 2e33aa436d984401993daed75b7a41b0 (DEFAULT_TABLE table=test-workload [id=122e1e087f99407c8276bb8c72ac7970]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260810 22:05:12.404924 998 tablet_service.cc:1467] Processing CreateTablet for tablet 33a9afb6e00543c184b1ef3bb975cc20 (DEFAULT_TABLE table=test-workload [id=122e1e087f99407c8276bb8c72ac7970]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260810 22:05:12.407001 997 tablet_service.cc:1467] Processing CreateTablet for tablet 92495f335e074d6fa01e66d395101407 (DEFAULT_TABLE table=test-workload [id=122e1e087f99407c8276bb8c72ac7970]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260810 22:05:12.409937 1000 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0192a67e9e45a78a45868b1ce6da9d (DEFAULT_TABLE table=test-workload [id=122e1e087f99407c8276bb8c72ac7970]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260810 22:05:12.399755 1605 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0192a67e9e45a78a45868b1ce6da9d (DEFAULT_TABLE table=test-workload [id=122e1e087f99407c8276bb8c72ac7970]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260810 22:05:12.401012 1603 tablet_service.cc:1467] Processing CreateTablet for tablet 632fa1d756704125b84b5e1772cb9076 (DEFAULT_TABLE table=test-workload [id=122e1e087f99407c8276bb8c72ac7970]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260810 22:05:12.402338 1602 tablet_service.cc:1467] Processing CreateTablet for tablet 33a9afb6e00543c184b1ef3bb975cc20 (DEFAULT_TABLE table=test-workload [id=122e1e087f99407c8276bb8c72ac7970]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260810 22:05:12.403687 1601 tablet_service.cc:1467] Processing CreateTablet for tablet 92495f335e074d6fa01e66d395101407 (DEFAULT_TABLE table=test-workload [id=122e1e087f99407c8276bb8c72ac7970]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260810 22:05:12.397228 1606 tablet_service.cc:1467] Processing CreateTablet for tablet 2e33aa436d984401993daed75b7a41b0 (DEFAULT_TABLE table=test-workload [id=122e1e087f99407c8276bb8c72ac7970]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260810 22:05:12.462673 1733 tablet_bootstrap.cc:492] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64: Bootstrap starting.
I20260810 22:05:12.466172 1727 tablet_bootstrap.cc:492] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64: Bootstrap starting.
I20260810 22:05:12.474792 1727 tablet_bootstrap.cc:654] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:12.477265 1727 log.cc:824] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64: Log is configured to *not* fsync() on all Append() calls
I20260810 22:05:12.479928 1733 tablet_bootstrap.cc:654] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:12.490840 1727 tablet_bootstrap.cc:492] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64: No bootstrap required, opened a new log
I20260810 22:05:12.491945 1727 ts_tablet_manager.cc:1397] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.003s
I20260810 22:05:12.502660 1733 tablet_bootstrap.cc:492] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64: No bootstrap required, opened a new log
I20260810 22:05:12.510924 1733 ts_tablet_manager.cc:1397] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.005s
I20260810 22:05:12.512327 1741 tablet_bootstrap.cc:492] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f: Bootstrap starting.
I20260810 22:05:12.523125 1738 tablet_bootstrap.cc:492] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: Bootstrap starting.
I20260810 22:05:12.534559 1728 tablet_bootstrap.cc:492] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64: Bootstrap starting.
I20260810 22:05:12.527058 1737 tablet_bootstrap.cc:492] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f: Bootstrap starting.
I20260810 22:05:12.543349 1742 tablet_bootstrap.cc:492] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: Bootstrap starting.
I20260810 22:05:12.544708 1743 tablet_bootstrap.cc:492] T 92495f335e074d6fa01e66d395101407 P 067bae0bd4a64f839e420a2f323862a5: Bootstrap starting.
I20260810 22:05:12.544485 1727 raft_consensus.cc:348] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.546641 1727 raft_consensus.cc:374] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:12.545898 1733 raft_consensus.cc:348] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.547132 1733 raft_consensus.cc:374] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:12.547452 1733 raft_consensus.cc:729] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912f44f41ac346ffa75962a42491ab64, State: Initialized, Role: FOLLOWER
I20260810 22:05:12.548594 1733 consensus_queue.cc:260] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.551026 1727 raft_consensus.cc:729] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912f44f41ac346ffa75962a42491ab64, State: Initialized, Role: FOLLOWER
I20260810 22:05:12.552039 1741 tablet_bootstrap.cc:654] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:12.552086 1743 tablet_bootstrap.cc:654] T 92495f335e074d6fa01e66d395101407 P 067bae0bd4a64f839e420a2f323862a5: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:12.551898 1727 consensus_queue.cc:260] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.554392 1743 log.cc:824] T 92495f335e074d6fa01e66d395101407 P 067bae0bd4a64f839e420a2f323862a5: Log is configured to *not* fsync() on all Append() calls
I20260810 22:05:12.555091 1738 tablet_bootstrap.cc:654] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:12.562462 1737 tablet_bootstrap.cc:654] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:12.566141 1743 tablet_bootstrap.cc:492] T 92495f335e074d6fa01e66d395101407 P 067bae0bd4a64f839e420a2f323862a5: No bootstrap required, opened a new log
I20260810 22:05:12.566730 1746 tablet_bootstrap.cc:492] T 4f0192a67e9e45a78a45868b1ce6da9d P 067bae0bd4a64f839e420a2f323862a5: Bootstrap starting.
I20260810 22:05:12.566867 1743 ts_tablet_manager.cc:1397] T 92495f335e074d6fa01e66d395101407 P 067bae0bd4a64f839e420a2f323862a5: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.004s
I20260810 22:05:12.566766 1738 log.cc:824] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: Log is configured to *not* fsync() on all Append() calls
I20260810 22:05:12.574900 1752 tablet_bootstrap.cc:492] T 632fa1d756704125b84b5e1772cb9076 P 067bae0bd4a64f839e420a2f323862a5: Bootstrap starting.
I20260810 22:05:12.576887 1742 tablet_bootstrap.cc:654] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:12.580385 1752 tablet_bootstrap.cc:654] T 632fa1d756704125b84b5e1772cb9076 P 067bae0bd4a64f839e420a2f323862a5: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:12.586876 1744 tablet_bootstrap.cc:492] T 33a9afb6e00543c184b1ef3bb975cc20 P 912f44f41ac346ffa75962a42491ab64: Bootstrap starting.
I20260810 22:05:12.590099 1756 tablet_bootstrap.cc:492] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: Bootstrap starting.
I20260810 22:05:12.599632 1728 tablet_bootstrap.cc:654] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:12.600775 1741 tablet_bootstrap.cc:492] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f: No bootstrap required, opened a new log
I20260810 22:05:12.604182 1741 ts_tablet_manager.cc:1397] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f: Time spent bootstrapping tablet: real 0.093s user 0.008s sys 0.013s
I20260810 22:05:12.606735 1742 tablet_bootstrap.cc:492] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: No bootstrap required, opened a new log
I20260810 22:05:12.610226 1742 ts_tablet_manager.cc:1397] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: Time spent bootstrapping tablet: real 0.067s user 0.006s sys 0.009s
I20260810 22:05:12.609795 1737 tablet_bootstrap.cc:492] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f: No bootstrap required, opened a new log
I20260810 22:05:12.613509 1737 ts_tablet_manager.cc:1397] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f: Time spent bootstrapping tablet: real 0.087s user 0.010s sys 0.008s
I20260810 22:05:12.609462 1738 tablet_bootstrap.cc:492] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: No bootstrap required, opened a new log
I20260810 22:05:12.616313 1738 ts_tablet_manager.cc:1397] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: Time spent bootstrapping tablet: real 0.094s user 0.014s sys 0.004s
I20260810 22:05:12.603756 1762 tablet_bootstrap.cc:492] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64: Bootstrap starting.
I20260810 22:05:12.611006 1733 ts_tablet_manager.cc:1428] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64: Time spent starting tablet: real 0.099s user 0.019s sys 0.014s
I20260810 22:05:12.639976 1737 raft_consensus.cc:348] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.641140 1737 raft_consensus.cc:374] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:12.641539 1737 raft_consensus.cc:729] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44427371e0c485eb96e803fd122ad5f, State: Initialized, Role: FOLLOWER
I20260810 22:05:12.642565 1737 consensus_queue.cc:260] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.643792 1756 tablet_bootstrap.cc:654] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:12.627928 1744 tablet_bootstrap.cc:654] T 33a9afb6e00543c184b1ef3bb975cc20 P 912f44f41ac346ffa75962a42491ab64: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:12.594540 1754 tablet_bootstrap.cc:492] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5: Bootstrap starting.
I20260810 22:05:12.602646 1743 raft_consensus.cc:348] T 92495f335e074d6fa01e66d395101407 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.645663 1743 raft_consensus.cc:374] T 92495f335e074d6fa01e66d395101407 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:12.645558 1742 raft_consensus.cc:348] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:12.629549 1763 raft_consensus.cc:482] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:12.646013 1743 raft_consensus.cc:729] T 92495f335e074d6fa01e66d395101407 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067bae0bd4a64f839e420a2f323862a5, State: Initialized, Role: FOLLOWER
I20260810 22:05:12.634586 1727 ts_tablet_manager.cc:1428] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64: Time spent starting tablet: real 0.142s user 0.030s sys 0.021s
I20260810 22:05:12.646377 1742 raft_consensus.cc:374] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:12.646667 1742 raft_consensus.cc:729] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44427371e0c485eb96e803fd122ad5f, State: Initialized, Role: FOLLOWER
I20260810 22:05:12.646618 1763 raft_consensus.cc:504] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.647128 1743 consensus_queue.cc:260] T 92495f335e074d6fa01e66d395101407 P 067bae0bd4a64f839e420a2f323862a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.647429 1742 consensus_queue.cc:260] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:12.653872 1738 raft_consensus.cc:348] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.657298 1738 raft_consensus.cc:374] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:12.660226 1738 raft_consensus.cc:729] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44427371e0c485eb96e803fd122ad5f, State: Initialized, Role: FOLLOWER
I20260810 22:05:12.662118 1738 consensus_queue.cc:260] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.658332 1741 raft_consensus.cc:348] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.667503 1741 raft_consensus.cc:374] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:12.668011 1741 raft_consensus.cc:729] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44427371e0c485eb96e803fd122ad5f, State: Initialized, Role: FOLLOWER
I20260810 22:05:12.671605 1741 consensus_queue.cc:260] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.608585 1746 tablet_bootstrap.cc:654] T 4f0192a67e9e45a78a45868b1ce6da9d P 067bae0bd4a64f839e420a2f323862a5: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:12.676775 1762 tablet_bootstrap.cc:654] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:12.683771 1728 tablet_bootstrap.cc:492] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64: No bootstrap required, opened a new log
I20260810 22:05:12.684214 1728 ts_tablet_manager.cc:1397] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64: Time spent bootstrapping tablet: real 0.150s user 0.009s sys 0.006s
I20260810 22:05:12.685456 1763 leader_election.cc:288] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991), 067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905)
I20260810 22:05:12.690924 1741 ts_tablet_manager.cc:1428] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f: Time spent starting tablet: real 0.086s user 0.019s sys 0.011s
I20260810 22:05:12.693078 1738 ts_tablet_manager.cc:1428] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: Time spent starting tablet: real 0.076s user 0.012s sys 0.003s
I20260810 22:05:12.682773 1742 ts_tablet_manager.cc:1428] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: Time spent starting tablet: real 0.072s user 0.008s sys 0.006s
I20260810 22:05:12.686434 1737 ts_tablet_manager.cc:1428] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f: Time spent starting tablet: real 0.073s user 0.012s sys 0.003s
I20260810 22:05:12.687289 1768 raft_consensus.cc:482] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:12.694977 1768 raft_consensus.cc:504] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.698727 1762 tablet_bootstrap.cc:492] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64: No bootstrap required, opened a new log
I20260810 22:05:12.700337 1762 ts_tablet_manager.cc:1397] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64: Time spent bootstrapping tablet: real 0.097s user 0.008s sys 0.007s
I20260810 22:05:12.701000 1768 leader_election.cc:288] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 912f44f41ac346ffa75962a42491ab64 (127.27.17.130:39307), 067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905)
I20260810 22:05:12.704880 1762 raft_consensus.cc:348] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.710111 1762 raft_consensus.cc:374] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:12.710865 1754 tablet_bootstrap.cc:654] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:12.710925 1762 raft_consensus.cc:729] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912f44f41ac346ffa75962a42491ab64, State: Initialized, Role: FOLLOWER
I20260810 22:05:12.714742 1766 tablet_bootstrap.cc:492] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5: Bootstrap starting.
I20260810 22:05:12.724663 1766 tablet_bootstrap.cc:654] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:12.727212 1695 heartbeater.cc:503] Master 127.27.17.190:37855 was elected leader, sending a full tablet report...
I20260810 22:05:12.725435 1752 tablet_bootstrap.cc:492] T 632fa1d756704125b84b5e1772cb9076 P 067bae0bd4a64f839e420a2f323862a5: No bootstrap required, opened a new log
I20260810 22:05:12.728184 1752 ts_tablet_manager.cc:1397] T 632fa1d756704125b84b5e1772cb9076 P 067bae0bd4a64f839e420a2f323862a5: Time spent bootstrapping tablet: real 0.153s user 0.008s sys 0.009s
I20260810 22:05:12.731225 1768 raft_consensus.cc:482] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:12.731707 1768 raft_consensus.cc:504] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:12.732116 1728 raft_consensus.cc:348] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:12.732932 1728 raft_consensus.cc:374] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:12.736696 1744 tablet_bootstrap.cc:492] T 33a9afb6e00543c184b1ef3bb975cc20 P 912f44f41ac346ffa75962a42491ab64: No bootstrap required, opened a new log
I20260810 22:05:12.737131 1744 ts_tablet_manager.cc:1397] T 33a9afb6e00543c184b1ef3bb975cc20 P 912f44f41ac346ffa75962a42491ab64: Time spent bootstrapping tablet: real 0.151s user 0.025s sys 0.015s
I20260810 22:05:12.733227 1728 raft_consensus.cc:729] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912f44f41ac346ffa75962a42491ab64, State: Initialized, Role: FOLLOWER
I20260810 22:05:12.739096 1746 tablet_bootstrap.cc:492] T 4f0192a67e9e45a78a45868b1ce6da9d P 067bae0bd4a64f839e420a2f323862a5: No bootstrap required, opened a new log
I20260810 22:05:12.739637 1746 ts_tablet_manager.cc:1397] T 4f0192a67e9e45a78a45868b1ce6da9d P 067bae0bd4a64f839e420a2f323862a5: Time spent bootstrapping tablet: real 0.173s user 0.008s sys 0.006s
I20260810 22:05:12.730935 1743 ts_tablet_manager.cc:1428] T 92495f335e074d6fa01e66d395101407 P 067bae0bd4a64f839e420a2f323862a5: Time spent starting tablet: real 0.164s user 0.054s sys 0.006s
I20260810 22:05:12.741477 1768 leader_election.cc:288] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 912f44f41ac346ffa75962a42491ab64 (127.27.17.130:39307), 067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905)
I20260810 22:05:12.742234 1768 raft_consensus.cc:482] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:12.743649 1768 raft_consensus.cc:504] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.744485 1752 raft_consensus.cc:348] T 632fa1d756704125b84b5e1772cb9076 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.746656 1752 raft_consensus.cc:374] T 632fa1d756704125b84b5e1772cb9076 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:12.747015 1752 raft_consensus.cc:729] T 632fa1d756704125b84b5e1772cb9076 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067bae0bd4a64f839e420a2f323862a5, State: Initialized, Role: FOLLOWER
I20260810 22:05:12.747800 1766 tablet_bootstrap.cc:492] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5: No bootstrap required, opened a new log
I20260810 22:05:12.748412 1768 leader_election.cc:288] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 912f44f41ac346ffa75962a42491ab64 (127.27.17.130:39307), 067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905)
I20260810 22:05:12.748644 1766 ts_tablet_manager.cc:1397] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.009s
I20260810 22:05:12.747927 1746 raft_consensus.cc:348] T 4f0192a67e9e45a78a45868b1ce6da9d P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.750010 1746 raft_consensus.cc:374] T 4f0192a67e9e45a78a45868b1ce6da9d P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:12.750356 1746 raft_consensus.cc:729] T 4f0192a67e9e45a78a45868b1ce6da9d P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067bae0bd4a64f839e420a2f323862a5, State: Initialized, Role: FOLLOWER
I20260810 22:05:12.747910 1752 consensus_queue.cc:260] T 632fa1d756704125b84b5e1772cb9076 P 067bae0bd4a64f839e420a2f323862a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.752050 1766 raft_consensus.cc:348] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:12.752784 1766 raft_consensus.cc:374] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:12.753062 1766 raft_consensus.cc:729] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067bae0bd4a64f839e420a2f323862a5, State: Initialized, Role: FOLLOWER
I20260810 22:05:12.753769 1766 consensus_queue.cc:260] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:12.755693 1756 tablet_bootstrap.cc:492] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: No bootstrap required, opened a new log
I20260810 22:05:12.755951 1766 ts_tablet_manager.cc:1428] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260810 22:05:12.712616 1762 consensus_queue.cc:260] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.757517 1756 ts_tablet_manager.cc:1397] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: Time spent bootstrapping tablet: real 0.168s user 0.015s sys 0.000s
I20260810 22:05:12.758262 1762 ts_tablet_manager.cc:1428] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64: Time spent starting tablet: real 0.057s user 0.008s sys 0.000s
I20260810 22:05:12.760334 1756 raft_consensus.cc:348] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.761178 1756 raft_consensus.cc:374] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:12.761476 1756 raft_consensus.cc:729] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44427371e0c485eb96e803fd122ad5f, State: Initialized, Role: FOLLOWER
I20260810 22:05:12.745250 1744 raft_consensus.cc:348] T 33a9afb6e00543c184b1ef3bb975cc20 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.762686 1744 raft_consensus.cc:374] T 33a9afb6e00543c184b1ef3bb975cc20 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:12.762993 1744 raft_consensus.cc:729] T 33a9afb6e00543c184b1ef3bb975cc20 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912f44f41ac346ffa75962a42491ab64, State: Initialized, Role: FOLLOWER
I20260810 22:05:12.763789 1744 consensus_queue.cc:260] T 33a9afb6e00543c184b1ef3bb975cc20 P 912f44f41ac346ffa75962a42491ab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.766422 1744 ts_tablet_manager.cc:1428] T 33a9afb6e00543c184b1ef3bb975cc20 P 912f44f41ac346ffa75962a42491ab64: Time spent starting tablet: real 0.029s user 0.006s sys 0.001s
I20260810 22:05:12.744323 1728 consensus_queue.cc:260] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:12.768656 1763 raft_consensus.cc:482] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:12.769088 1763 raft_consensus.cc:504] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.770308 1754 tablet_bootstrap.cc:492] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5: No bootstrap required, opened a new log
I20260810 22:05:12.770779 1754 ts_tablet_manager.cc:1397] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5: Time spent bootstrapping tablet: real 0.176s user 0.013s sys 0.007s
I20260810 22:05:12.770947 1763 leader_election.cc:288] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991), 067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905)
I20260810 22:05:12.773476 1754 raft_consensus.cc:348] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.774243 1754 raft_consensus.cc:374] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:12.774539 1754 raft_consensus.cc:729] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067bae0bd4a64f839e420a2f323862a5, State: Initialized, Role: FOLLOWER
I20260810 22:05:12.775224 1754 consensus_queue.cc:260] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.777626 1754 ts_tablet_manager.cc:1428] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 22:05:12.778266 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92495f335e074d6fa01e66d395101407" candidate_uuid: "912f44f41ac346ffa75962a42491ab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067bae0bd4a64f839e420a2f323862a5" is_pre_election: true
I20260810 22:05:12.764647 1756 consensus_queue.cc:260] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.779309 1636 raft_consensus.cc:2454] T 92495f335e074d6fa01e66d395101407 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 912f44f41ac346ffa75962a42491ab64 in term 0.
I20260810 22:05:12.780063 1728 ts_tablet_manager.cc:1428] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64: Time spent starting tablet: real 0.095s user 0.007s sys 0.000s
I20260810 22:05:12.780786 1635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0192a67e9e45a78a45868b1ce6da9d" candidate_uuid: "912f44f41ac346ffa75962a42491ab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067bae0bd4a64f839e420a2f323862a5" is_pre_election: true
I20260810 22:05:12.782419 1756 ts_tablet_manager.cc:1428] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20260810 22:05:12.782892 1240 leader_election.cc:302] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067bae0bd4a64f839e420a2f323862a5, 912f44f41ac346ffa75962a42491ab64; no voters:
I20260810 22:05:12.785238 1763 raft_consensus.cc:2781] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:12.784967 1339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632fa1d756704125b84b5e1772cb9076" candidate_uuid: "e44427371e0c485eb96e803fd122ad5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912f44f41ac346ffa75962a42491ab64" is_pre_election: true
I20260810 22:05:12.785660 1763 raft_consensus.cc:482] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:12.785741 1336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0192a67e9e45a78a45868b1ce6da9d" candidate_uuid: "e44427371e0c485eb96e803fd122ad5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912f44f41ac346ffa75962a42491ab64" is_pre_election: true
I20260810 22:05:12.786034 1763 raft_consensus.cc:3037] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:12.785995 1339 raft_consensus.cc:2454] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e44427371e0c485eb96e803fd122ad5f in term 0.
I20260810 22:05:12.786405 1336 raft_consensus.cc:2454] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e44427371e0c485eb96e803fd122ad5f in term 0.
I20260810 22:05:12.787266 1337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e33aa436d984401993daed75b7a41b0" candidate_uuid: "e44427371e0c485eb96e803fd122ad5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912f44f41ac346ffa75962a42491ab64" is_pre_election: true
I20260810 22:05:12.787707 1029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92495f335e074d6fa01e66d395101407" candidate_uuid: "912f44f41ac346ffa75962a42491ab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e44427371e0c485eb96e803fd122ad5f" is_pre_election: true
I20260810 22:05:12.788030 1337 raft_consensus.cc:2454] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e44427371e0c485eb96e803fd122ad5f in term 0.
I20260810 22:05:12.788576 1029 raft_consensus.cc:2454] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 912f44f41ac346ffa75962a42491ab64 in term 0.
I20260810 22:05:12.790030 1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0192a67e9e45a78a45868b1ce6da9d" candidate_uuid: "912f44f41ac346ffa75962a42491ab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e44427371e0c485eb96e803fd122ad5f" is_pre_election: true
I20260810 22:05:12.790720 1027 raft_consensus.cc:2454] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 912f44f41ac346ffa75962a42491ab64 in term 0.
I20260810 22:05:12.790899 1752 ts_tablet_manager.cc:1428] T 632fa1d756704125b84b5e1772cb9076 P 067bae0bd4a64f839e420a2f323862a5: Time spent starting tablet: real 0.060s user 0.001s sys 0.006s
I20260810 22:05:12.754184 1746 consensus_queue.cc:260] T 4f0192a67e9e45a78a45868b1ce6da9d P 067bae0bd4a64f839e420a2f323862a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.792693 1247 leader_election.cc:302] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 912f44f41ac346ffa75962a42491ab64, e44427371e0c485eb96e803fd122ad5f; no voters:
I20260810 22:05:12.792737 1635 raft_consensus.cc:2454] T 4f0192a67e9e45a78a45868b1ce6da9d P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 912f44f41ac346ffa75962a42491ab64 in term 0.
I20260810 22:05:12.793663 1745 raft_consensus.cc:2781] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:12.794036 1745 raft_consensus.cc:482] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:12.794381 1745 raft_consensus.cc:3037] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:12.794443 1746 ts_tablet_manager.cc:1428] T 4f0192a67e9e45a78a45868b1ce6da9d P 067bae0bd4a64f839e420a2f323862a5: Time spent starting tablet: real 0.054s user 0.004s sys 0.002s
I20260810 22:05:12.797613 1763 raft_consensus.cc:504] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.798349 942 leader_election.cc:302] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 912f44f41ac346ffa75962a42491ab64, e44427371e0c485eb96e803fd122ad5f; no voters:
I20260810 22:05:12.799441 942 leader_election.cc:302] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 912f44f41ac346ffa75962a42491ab64, e44427371e0c485eb96e803fd122ad5f; no voters:
I20260810 22:05:12.800431 1781 raft_consensus.cc:2781] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:12.800393 942 leader_election.cc:302] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 912f44f41ac346ffa75962a42491ab64, e44427371e0c485eb96e803fd122ad5f; no voters:
I20260810 22:05:12.800882 1781 raft_consensus.cc:482] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:12.801260 1768 raft_consensus.cc:2781] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:12.801656 1783 raft_consensus.cc:2781] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:12.802059 1783 raft_consensus.cc:482] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:12.802402 1783 raft_consensus.cc:3037] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:12.804836 1745 raft_consensus.cc:504] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.801674 1768 raft_consensus.cc:482] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:12.805963 1768 raft_consensus.cc:3037] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:12.809118 1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92495f335e074d6fa01e66d395101407" candidate_uuid: "912f44f41ac346ffa75962a42491ab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e44427371e0c485eb96e803fd122ad5f"
I20260810 22:05:12.801282 1781 raft_consensus.cc:3037] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:12.809762 1027 raft_consensus.cc:3037] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:12.810175 1745 leader_election.cc:288] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 1 election: Requested vote from peers e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991), 067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905)
I20260810 22:05:12.810829 1763 leader_election.cc:288] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 1 election: Requested vote from peers e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991), 067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905)
I20260810 22:05:12.811866 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92495f335e074d6fa01e66d395101407" candidate_uuid: "912f44f41ac346ffa75962a42491ab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067bae0bd4a64f839e420a2f323862a5"
I20260810 22:05:12.811968 1635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0192a67e9e45a78a45868b1ce6da9d" candidate_uuid: "912f44f41ac346ffa75962a42491ab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067bae0bd4a64f839e420a2f323862a5"
I20260810 22:05:12.812556 1636 raft_consensus.cc:3037] T 92495f335e074d6fa01e66d395101407 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:12.812686 1635 raft_consensus.cc:3037] T 4f0192a67e9e45a78a45868b1ce6da9d P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:12.818444 1029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0192a67e9e45a78a45868b1ce6da9d" candidate_uuid: "912f44f41ac346ffa75962a42491ab64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e44427371e0c485eb96e803fd122ad5f"
I20260810 22:05:12.819801 1636 raft_consensus.cc:2454] T 92495f335e074d6fa01e66d395101407 P 067bae0bd4a64f839e420a2f323862a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912f44f41ac346ffa75962a42491ab64 in term 1.
I20260810 22:05:12.819742 1635 raft_consensus.cc:2454] T 4f0192a67e9e45a78a45868b1ce6da9d P 067bae0bd4a64f839e420a2f323862a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912f44f41ac346ffa75962a42491ab64 in term 1.
I20260810 22:05:12.821043 1240 leader_election.cc:302] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067bae0bd4a64f839e420a2f323862a5, 912f44f41ac346ffa75962a42491ab64; no voters:
I20260810 22:05:12.821990 1763 raft_consensus.cc:2781] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:12.822072 1240 leader_election.cc:302] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067bae0bd4a64f839e420a2f323862a5, 912f44f41ac346ffa75962a42491ab64; no voters:
I20260810 22:05:12.822701 1763 raft_consensus.cc:686] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 [term 1 LEADER]: Becoming Leader. State: Replica: 912f44f41ac346ffa75962a42491ab64, State: Running, Role: LEADER
I20260810 22:05:12.823202 1750 raft_consensus.cc:2781] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:12.823683 1750 raft_consensus.cc:686] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [term 1 LEADER]: Becoming Leader. State: Replica: 912f44f41ac346ffa75962a42491ab64, State: Running, Role: LEADER
I20260810 22:05:12.823901 1763 consensus_queue.cc:237] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.824503 1750 consensus_queue.cc:237] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.829551 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e33aa436d984401993daed75b7a41b0" candidate_uuid: "e44427371e0c485eb96e803fd122ad5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067bae0bd4a64f839e420a2f323862a5" is_pre_election: true
I20260810 22:05:12.829458 1783 raft_consensus.cc:504] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:12.830227 1636 raft_consensus.cc:2454] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e44427371e0c485eb96e803fd122ad5f in term 0.
I20260810 22:05:12.830140 1634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0192a67e9e45a78a45868b1ce6da9d" candidate_uuid: "e44427371e0c485eb96e803fd122ad5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067bae0bd4a64f839e420a2f323862a5" is_pre_election: true
I20260810 22:05:12.831099 1634 raft_consensus.cc:2379] T 4f0192a67e9e45a78a45868b1ce6da9d P 067bae0bd4a64f839e420a2f323862a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e44427371e0c485eb96e803fd122ad5f in current term 1: Already voted for candidate 912f44f41ac346ffa75962a42491ab64 in this term.
I20260810 22:05:12.832633 1634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e33aa436d984401993daed75b7a41b0" candidate_uuid: "e44427371e0c485eb96e803fd122ad5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067bae0bd4a64f839e420a2f323862a5"
I20260810 22:05:12.833025 1635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632fa1d756704125b84b5e1772cb9076" candidate_uuid: "e44427371e0c485eb96e803fd122ad5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067bae0bd4a64f839e420a2f323862a5" is_pre_election: true
I20260810 22:05:12.833176 1634 raft_consensus.cc:3037] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:12.833618 1635 raft_consensus.cc:2454] T 632fa1d756704125b84b5e1772cb9076 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e44427371e0c485eb96e803fd122ad5f in term 0.
I20260810 22:05:12.834074 1337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e33aa436d984401993daed75b7a41b0" candidate_uuid: "e44427371e0c485eb96e803fd122ad5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912f44f41ac346ffa75962a42491ab64"
I20260810 22:05:12.834757 1337 raft_consensus.cc:3037] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Advancing to term 1
W20260810 22:05:12.837723 1097 tablet.cc:2369] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:05:12.838833 1027 raft_consensus.cc:2454] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912f44f41ac346ffa75962a42491ab64 in term 1.
I20260810 22:05:12.835646 1768 raft_consensus.cc:504] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.841094 1634 raft_consensus.cc:2454] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e44427371e0c485eb96e803fd122ad5f in term 1.
I20260810 22:05:12.843014 1029 raft_consensus.cc:2379] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 912f44f41ac346ffa75962a42491ab64 in current term 1: Already voted for candidate e44427371e0c485eb96e803fd122ad5f in this term.
I20260810 22:05:12.843093 941 leader_election.cc:302] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067bae0bd4a64f839e420a2f323862a5, e44427371e0c485eb96e803fd122ad5f; no voters:
I20260810 22:05:12.844055 1772 raft_consensus.cc:2781] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:12.846639 1772 raft_consensus.cc:686] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [term 1 LEADER]: Becoming Leader. State: Replica: e44427371e0c485eb96e803fd122ad5f, State: Running, Role: LEADER
I20260810 22:05:12.847000 1781 raft_consensus.cc:504] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.847714 1772 consensus_queue.cc:237] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:12.850162 1635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0192a67e9e45a78a45868b1ce6da9d" candidate_uuid: "e44427371e0c485eb96e803fd122ad5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067bae0bd4a64f839e420a2f323862a5"
I20260810 22:05:12.848773 1768 leader_election.cc:288] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [CANDIDATE]: Term 1 election: Requested vote from peers 912f44f41ac346ffa75962a42491ab64 (127.27.17.130:39307), 067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905)
I20260810 22:05:12.850940 1781 leader_election.cc:288] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [CANDIDATE]: Term 1 election: Requested vote from peers 912f44f41ac346ffa75962a42491ab64 (127.27.17.130:39307), 067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905)
I20260810 22:05:12.851549 1635 raft_consensus.cc:2379] T 4f0192a67e9e45a78a45868b1ce6da9d P 067bae0bd4a64f839e420a2f323862a5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e44427371e0c485eb96e803fd122ad5f in current term 1: Already voted for candidate 912f44f41ac346ffa75962a42491ab64 in this term.
I20260810 22:05:12.843107 1783 leader_election.cc:288] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [CANDIDATE]: Term 1 election: Requested vote from peers 912f44f41ac346ffa75962a42491ab64 (127.27.17.130:39307), 067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905)
I20260810 22:05:12.853802 1339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632fa1d756704125b84b5e1772cb9076" candidate_uuid: "e44427371e0c485eb96e803fd122ad5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912f44f41ac346ffa75962a42491ab64"
I20260810 22:05:12.854561 1339 raft_consensus.cc:3037] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Advancing to term 1
W20260810 22:05:12.857242 1398 tablet.cc:2369] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:05:12.853816 1336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0192a67e9e45a78a45868b1ce6da9d" candidate_uuid: "e44427371e0c485eb96e803fd122ad5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912f44f41ac346ffa75962a42491ab64"
I20260810 22:05:12.859540 942 leader_election.cc:302] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e44427371e0c485eb96e803fd122ad5f; no voters: 067bae0bd4a64f839e420a2f323862a5, 912f44f41ac346ffa75962a42491ab64
I20260810 22:05:12.860411 1783 raft_consensus.cc:2726] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 22:05:12.861774 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632fa1d756704125b84b5e1772cb9076" candidate_uuid: "e44427371e0c485eb96e803fd122ad5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067bae0bd4a64f839e420a2f323862a5"
I20260810 22:05:12.862341 1636 raft_consensus.cc:3037] T 632fa1d756704125b84b5e1772cb9076 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:12.854439 1337 raft_consensus.cc:2454] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e44427371e0c485eb96e803fd122ad5f in term 1.
I20260810 22:05:12.867648 1339 raft_consensus.cc:2454] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e44427371e0c485eb96e803fd122ad5f in term 1.
I20260810 22:05:12.868592 1636 raft_consensus.cc:2454] T 632fa1d756704125b84b5e1772cb9076 P 067bae0bd4a64f839e420a2f323862a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e44427371e0c485eb96e803fd122ad5f in term 1.
I20260810 22:05:12.868886 942 leader_election.cc:302] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 912f44f41ac346ffa75962a42491ab64, e44427371e0c485eb96e803fd122ad5f; no voters:
I20260810 22:05:12.869650 1783 raft_consensus.cc:2781] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:12.870095 1783 raft_consensus.cc:686] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [term 1 LEADER]: Becoming Leader. State: Replica: e44427371e0c485eb96e803fd122ad5f, State: Running, Role: LEADER
I20260810 22:05:12.870967 1783 consensus_queue.cc:237] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.875908 721 catalog_manager.cc:5495] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 912f44f41ac346ffa75962a42491ab64 (127.27.17.130). New cstate: current_term: 1 leader_uuid: "912f44f41ac346ffa75962a42491ab64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } }
I20260810 22:05:12.880385 716 catalog_manager.cc:5495] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f reported cstate change: term changed from 0 to 1, leader changed from <none> to e44427371e0c485eb96e803fd122ad5f (127.27.17.129). New cstate: current_term: 1 leader_uuid: "e44427371e0c485eb96e803fd122ad5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } }
I20260810 22:05:12.877836 721 catalog_manager.cc:5495] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 912f44f41ac346ffa75962a42491ab64 (127.27.17.130). New cstate: current_term: 1 leader_uuid: "912f44f41ac346ffa75962a42491ab64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } }
I20260810 22:05:12.883606 1770 raft_consensus.cc:482] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:12.884184 1770 raft_consensus.cc:504] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.888949 1770 leader_election.cc:288] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991), 912f44f41ac346ffa75962a42491ab64 (127.27.17.130:39307)
I20260810 22:05:12.920328 1029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a9afb6e00543c184b1ef3bb975cc20" candidate_uuid: "067bae0bd4a64f839e420a2f323862a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e44427371e0c485eb96e803fd122ad5f" is_pre_election: true
I20260810 22:05:12.921203 1029 raft_consensus.cc:2454] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 067bae0bd4a64f839e420a2f323862a5 in term 0.
I20260810 22:05:12.920831 716 catalog_manager.cc:5495] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f reported cstate change: term changed from 0 to 1, leader changed from <none> to e44427371e0c485eb96e803fd122ad5f (127.27.17.129). New cstate: current_term: 1 leader_uuid: "e44427371e0c485eb96e803fd122ad5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } }
I20260810 22:05:12.922869 1545 leader_election.cc:302] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067bae0bd4a64f839e420a2f323862a5, e44427371e0c485eb96e803fd122ad5f; no voters:
I20260810 22:05:12.923815 1770 raft_consensus.cc:2781] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:12.924181 1770 raft_consensus.cc:482] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:12.924539 1770 raft_consensus.cc:3037] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:12.924917 1339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a9afb6e00543c184b1ef3bb975cc20" candidate_uuid: "067bae0bd4a64f839e420a2f323862a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912f44f41ac346ffa75962a42491ab64" is_pre_election: true
I20260810 22:05:12.925539 1339 raft_consensus.cc:2454] T 33a9afb6e00543c184b1ef3bb975cc20 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 067bae0bd4a64f839e420a2f323862a5 in term 0.
I20260810 22:05:12.931471 1770 raft_consensus.cc:504] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.933095 1770 leader_election.cc:288] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [CANDIDATE]: Term 1 election: Requested vote from peers e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991), 912f44f41ac346ffa75962a42491ab64 (127.27.17.130:39307)
I20260810 22:05:12.934024 1029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a9afb6e00543c184b1ef3bb975cc20" candidate_uuid: "067bae0bd4a64f839e420a2f323862a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e44427371e0c485eb96e803fd122ad5f"
I20260810 22:05:12.934211 1339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a9afb6e00543c184b1ef3bb975cc20" candidate_uuid: "067bae0bd4a64f839e420a2f323862a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912f44f41ac346ffa75962a42491ab64"
I20260810 22:05:12.934517 1029 raft_consensus.cc:3037] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:12.934816 1339 raft_consensus.cc:3037] T 33a9afb6e00543c184b1ef3bb975cc20 P 912f44f41ac346ffa75962a42491ab64 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:12.939401 1029 raft_consensus.cc:2454] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 067bae0bd4a64f839e420a2f323862a5 in term 1.
I20260810 22:05:12.940505 1545 leader_election.cc:302] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067bae0bd4a64f839e420a2f323862a5, e44427371e0c485eb96e803fd122ad5f; no voters:
I20260810 22:05:12.941010 1339 raft_consensus.cc:2454] T 33a9afb6e00543c184b1ef3bb975cc20 P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 067bae0bd4a64f839e420a2f323862a5 in term 1.
I20260810 22:05:12.941408 1770 raft_consensus.cc:2781] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:12.944082 1770 raft_consensus.cc:686] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [term 1 LEADER]: Becoming Leader. State: Replica: 067bae0bd4a64f839e420a2f323862a5, State: Running, Role: LEADER
I20260810 22:05:12.944931 1770 consensus_queue.cc:237] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:12.955837 716 catalog_manager.cc:5495] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 067bae0bd4a64f839e420a2f323862a5 (127.27.17.131). New cstate: current_term: 1 leader_uuid: "067bae0bd4a64f839e420a2f323862a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } }
W20260810 22:05:12.956699 1696 tablet.cc:2369] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:05:13.028494 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 806
I20260810 22:05:13.086585 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.129:32991
--local_ip_for_outbound_sockets=127.27.17.129
--tserver_master_addrs=127.27.17.190:37855
--webserver_port=33191
--webserver_interface=127.27.17.129
--env_inject_eio_globs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-1/**
--env_inject_eio=1.0
--crash_on_eio=false
--builtin_ntp_servers=127.27.17.148:37545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20260810 22:05:13.207181 1763 consensus_queue.cc:1035] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 22:05:13.228581 1247 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.129:32991: connect: Connection refused (error 111)
W20260810 22:05:13.232920 1247 consensus_peers.cc:487] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 -> Peer e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Couldn't send request to peer e44427371e0c485eb96e803fd122ad5f. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:32991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 22:05:13.259318 1247 consensus_peers.cc:487] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 -> Peer e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Couldn't send request to peer e44427371e0c485eb96e803fd122ad5f. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:32991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 22:05:13.272995 1750 consensus_queue.cc:1035] T 4f0192a67e9e45a78a45868b1ce6da9d P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 22:05:13.434296 1819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:05:13.434841 1819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:05:13.435525 1819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 22:05:13.442150 1770 consensus_queue.cc:1035] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 22:05:13.457840 1545 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.129:32991: connect: Connection refused (error 111)
W20260810 22:05:13.461849 1545 consensus_peers.cc:487] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 -> Peer e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Couldn't send request to peer e44427371e0c485eb96e803fd122ad5f. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:32991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 22:05:13.475836 1819 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260810 22:05:13.476197 1819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:05:13.476756 1819 flags.cc:425] Enabled experimental flag: --crash_on_eio=false
W20260810 22:05:13.477202 1819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.129
I20260810 22:05:13.512444 1819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:37545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.129:32991
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.27.17.129
--webserver_port=33191
--tserver_master_addrs=127.27.17.190:37855
--env_inject_eio=1
--env_inject_eio_globs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-1/**
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.129
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:05:13.513686 1819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260810 22:05:13.514461 1819 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260810 22:05:13.514624 1819 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260810 22:05:13.515535 1819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:05:13.534413 1837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:14.374452 1745 raft_consensus.cc:482] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:14.374809 1745 raft_consensus.cc:504] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:14.376289 1745 leader_election.cc:288] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991), 067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905)
I20260810 22:05:14.377318 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632fa1d756704125b84b5e1772cb9076" candidate_uuid: "912f44f41ac346ffa75962a42491ab64" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067bae0bd4a64f839e420a2f323862a5" is_pre_election: true
I20260810 22:05:14.377960 1636 raft_consensus.cc:2454] T 632fa1d756704125b84b5e1772cb9076 P 067bae0bd4a64f839e420a2f323862a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 912f44f41ac346ffa75962a42491ab64 in term 1.
I20260810 22:05:14.378995 1240 leader_election.cc:302] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067bae0bd4a64f839e420a2f323862a5, 912f44f41ac346ffa75962a42491ab64; no voters:
I20260810 22:05:14.379704 1745 raft_consensus.cc:2781] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 22:05:14.380198 1745 raft_consensus.cc:482] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:14.380622 1745 raft_consensus.cc:3037] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Advancing to term 2
W20260810 22:05:14.380782 1247 leader_election.cc:334] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Network error: Client connection negotiation failed: client connection to 127.27.17.129:32991: connect: Connection refused (error 111)
I20260810 22:05:14.390213 1745 raft_consensus.cc:504] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:14.392985 1745 leader_election.cc:288] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 2 election: Requested vote from peers e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991), 067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905)
I20260810 22:05:14.394204 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632fa1d756704125b84b5e1772cb9076" candidate_uuid: "912f44f41ac346ffa75962a42491ab64" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067bae0bd4a64f839e420a2f323862a5"
I20260810 22:05:14.394708 1636 raft_consensus.cc:3037] T 632fa1d756704125b84b5e1772cb9076 P 067bae0bd4a64f839e420a2f323862a5 [term 1 FOLLOWER]: Advancing to term 2
W20260810 22:05:14.397053 1247 leader_election.cc:334] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Network error: Client connection negotiation failed: client connection to 127.27.17.129:32991: connect: Connection refused (error 111)
I20260810 22:05:14.399561 1636 raft_consensus.cc:2454] T 632fa1d756704125b84b5e1772cb9076 P 067bae0bd4a64f839e420a2f323862a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912f44f41ac346ffa75962a42491ab64 in term 2.
I20260810 22:05:14.400488 1240 leader_election.cc:302] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 067bae0bd4a64f839e420a2f323862a5, 912f44f41ac346ffa75962a42491ab64; no voters: e44427371e0c485eb96e803fd122ad5f
I20260810 22:05:14.401224 1745 raft_consensus.cc:2781] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:05:14.401939 1745 raft_consensus.cc:686] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [term 2 LEADER]: Becoming Leader. State: Replica: 912f44f41ac346ffa75962a42491ab64, State: Running, Role: LEADER
I20260810 22:05:14.403071 1745 consensus_queue.cc:237] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:14.413460 721 catalog_manager.cc:5495] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 reported cstate change: term changed from 1 to 2, leader changed from e44427371e0c485eb96e803fd122ad5f (127.27.17.129) to 912f44f41ac346ffa75962a42491ab64 (127.27.17.130). New cstate: current_term: 2 leader_uuid: "912f44f41ac346ffa75962a42491ab64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } }
I20260810 22:05:14.764856 1745 raft_consensus.cc:482] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:14.765343 1745 raft_consensus.cc:504] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:14.766978 1745 leader_election.cc:288] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905), e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991)
I20260810 22:05:14.767957 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e33aa436d984401993daed75b7a41b0" candidate_uuid: "912f44f41ac346ffa75962a42491ab64" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067bae0bd4a64f839e420a2f323862a5" is_pre_election: true
I20260810 22:05:14.768646 1636 raft_consensus.cc:2454] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 912f44f41ac346ffa75962a42491ab64 in term 1.
I20260810 22:05:14.769786 1240 leader_election.cc:302] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067bae0bd4a64f839e420a2f323862a5, 912f44f41ac346ffa75962a42491ab64; no voters:
I20260810 22:05:14.770572 1745 raft_consensus.cc:2781] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260810 22:05:14.770772 1247 leader_election.cc:334] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Network error: Client connection negotiation failed: client connection to 127.27.17.129:32991: connect: Connection refused (error 111)
I20260810 22:05:14.770911 1745 raft_consensus.cc:482] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:14.771237 1745 raft_consensus.cc:3037] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:05:14.775790 1745 raft_consensus.cc:504] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:14.777168 1745 leader_election.cc:288] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 2 election: Requested vote from peers 067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905), e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991)
I20260810 22:05:14.778126 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e33aa436d984401993daed75b7a41b0" candidate_uuid: "912f44f41ac346ffa75962a42491ab64" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067bae0bd4a64f839e420a2f323862a5"
I20260810 22:05:14.778600 1636 raft_consensus.cc:3037] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5 [term 1 FOLLOWER]: Advancing to term 2
W20260810 22:05:14.780606 1247 leader_election.cc:334] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Network error: Client connection negotiation failed: client connection to 127.27.17.129:32991: connect: Connection refused (error 111)
I20260810 22:05:14.784693 1636 raft_consensus.cc:2454] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912f44f41ac346ffa75962a42491ab64 in term 2.
I20260810 22:05:14.785877 1240 leader_election.cc:302] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 067bae0bd4a64f839e420a2f323862a5, 912f44f41ac346ffa75962a42491ab64; no voters: e44427371e0c485eb96e803fd122ad5f
I20260810 22:05:14.786530 1745 raft_consensus.cc:2781] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:05:14.787009 1745 raft_consensus.cc:686] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [term 2 LEADER]: Becoming Leader. State: Replica: 912f44f41ac346ffa75962a42491ab64, State: Running, Role: LEADER
I20260810 22:05:14.787878 1745 consensus_queue.cc:237] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:14.795054 721 catalog_manager.cc:5495] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 reported cstate change: term changed from 1 to 2, leader changed from e44427371e0c485eb96e803fd122ad5f (127.27.17.129) to 912f44f41ac346ffa75962a42491ab64 (127.27.17.130). New cstate: current_term: 2 leader_uuid: "912f44f41ac346ffa75962a42491ab64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } }
W20260810 22:05:13.534467 1840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:13.538094 1843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:13.539619 1819 server_base.cc:1034] running on GCE node
I20260810 22:05:14.804507 1819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:05:14.807122 1819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:05:14.808694 1819 hybrid_clock.cc:648] HybridClock initialized: now 1786399514808640 us; error 59 us; skew 500 ppm
I20260810 22:05:14.809684 1819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:05:14.816836 1819 webserver.cc:460] Webserver started at http://127.27.17.129:33191/ using document root <none> and password file <none>
I20260810 22:05:14.818676 1819 fs_manager.cc:359] Metadata directory not provided
I20260810 22:05:14.818958 1819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
E20260810 22:05:14.821874 1819 env_posix.cc:432] I/O error, context: INJECTED FAILURE
W20260810 22:05:14.827422 1247 consensus_peers.cc:487] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 -> Peer e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Couldn't send request to peer e44427371e0c485eb96e803fd122ad5f. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:32991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
E20260810 22:05:14.829178 1819 env_posix.cc:432] I/O error, context: INJECTED FAILURE
E20260810 22:05:14.846144 1819 env_posix.cc:432] I/O error, context: INJECTED FAILURE
I20260810 22:05:14.859591 1819 fs_manager.cc:709] Time spent opening directory manager: real 0.033s user 0.027s sys 0.004s
I20260810 22:05:14.978205 1636 raft_consensus.cc:1261] T 632fa1d756704125b84b5e1772cb9076 P 067bae0bd4a64f839e420a2f323862a5 [term 2 FOLLOWER]: Refusing update from remote peer 912f44f41ac346ffa75962a42491ab64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260810 22:05:14.980427 1745 consensus_queue.cc:1035] T 632fa1d756704125b84b5e1772cb9076 P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
E20260810 22:05:15.289394 1819 log_block_manager.cc:2924] Not using report from /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-1/data: IO error: Data directory failed (error 5)
I20260810 22:05:15.291733 1819 fs_manager.cc:725] Time spent opening block manager: real 0.429s user 0.004s sys 0.000s
I20260810 22:05:15.292053 1819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/wal
uuid: "e44427371e0c485eb96e803fd122ad5f"
format_stamp: "Formatted at 2026-08-10 22:05:06 on dist-test-slave-bw88"
I20260810 22:05:15.292567 1819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/wal
4 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260810 22:05:15.302564 1247 consensus_peers.cc:487] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 -> Peer e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Couldn't send request to peer e44427371e0c485eb96e803fd122ad5f. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:32991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 22:05:15.335419 1819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:05:15.337505 1819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:05:15.338085 1819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:05:15.340929 1819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:05:15.346777 1969 ts_tablet_manager.cc:542] Loading tablet metadata (0/5 complete)
W20260810 22:05:15.358965 1969 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet 2e33aa436d984401993daed75b7a41b0: could not find data dir with uuid 07c75217e2fa4b7db14d05ece09f0347
W20260810 22:05:15.359433 1970 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet 33a9afb6e00543c184b1ef3bb975cc20: could not find data dir with uuid 07c75217e2fa4b7db14d05ece09f0347
W20260810 22:05:15.368201 1970 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet 92495f335e074d6fa01e66d395101407: could not find data dir with uuid 07c75217e2fa4b7db14d05ece09f0347
I20260810 22:05:15.368788 1819 ts_tablet_manager.cc:579] Loaded tablet metadata (5 total tablets, 5 live tablets)
I20260810 22:05:15.369038 1819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.024s user 0.013s sys 0.007s
I20260810 22:05:15.369318 1819 ts_tablet_manager.cc:594] Registering tablets (0/5 complete)
I20260810 22:05:15.371502 1634 raft_consensus.cc:1261] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5 [term 2 FOLLOWER]: Refusing update from remote peer 912f44f41ac346ffa75962a42491ab64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260810 22:05:15.372738 1848 consensus_queue.cc:1035] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:15.377058 1970 tablet_bootstrap.cc:492] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: Bootstrap starting.
E20260810 22:05:15.377691 1970 ts_tablet_manager.cc:1419] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 2e33aa436d984401993daed75b7a41b0
I20260810 22:05:15.378086 1970 ts_tablet_manager.cc:1397] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:05:15.378404 1970 tablet_replica.cc:330] stopping tablet replica
I20260810 22:05:15.378755 1970 raft_consensus.cc:2229] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:05:15.382347 1972 tablet_bootstrap.cc:492] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: Bootstrap starting.
I20260810 22:05:15.382586 1970 raft_consensus.cc:2258] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Raft consensus is shut down!
E20260810 22:05:15.382807 1972 ts_tablet_manager.cc:1419] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 33a9afb6e00543c184b1ef3bb975cc20
I20260810 22:05:15.383070 1972 ts_tablet_manager.cc:1397] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:05:15.383355 1972 tablet_replica.cc:330] stopping tablet replica
I20260810 22:05:15.383571 1972 raft_consensus.cc:2229] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:05:15.383855 1972 raft_consensus.cc:2258] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:05:15.390226 1972 tablet_bootstrap.cc:492] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f: Bootstrap starting.
I20260810 22:05:15.395277 1970 tablet_bootstrap.cc:492] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f: Bootstrap starting.
I20260810 22:05:15.399119 1819 ts_tablet_manager.cc:610] Registered 5 tablets
I20260810 22:05:15.399394 1819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.030s user 0.016s sys 0.011s
I20260810 22:05:15.399631 1973 tablet_bootstrap.cc:492] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: Bootstrap starting.
E20260810 22:05:15.400094 1973 ts_tablet_manager.cc:1419] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 92495f335e074d6fa01e66d395101407
I20260810 22:05:15.400404 1973 ts_tablet_manager.cc:1397] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:05:15.400694 1973 tablet_replica.cc:330] stopping tablet replica
I20260810 22:05:15.400991 1973 raft_consensus.cc:2229] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:05:15.401417 1973 raft_consensus.cc:2258] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:05:15.456581 1970 log.cc:824] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f: Log is configured to *not* fsync() on all Append() calls
I20260810 22:05:15.465118 1970 tablet_bootstrap.cc:492] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:05:15.466499 1970 tablet_bootstrap.cc:492] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f: Bootstrap complete.
I20260810 22:05:15.468616 1970 ts_tablet_manager.cc:1397] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f: Time spent bootstrapping tablet: real 0.074s user 0.038s sys 0.033s
I20260810 22:05:15.475683 1972 tablet_bootstrap.cc:492] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260810 22:05:15.477294 1972 tablet_bootstrap.cc:492] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f: Bootstrap complete.
I20260810 22:05:15.479442 1972 ts_tablet_manager.cc:1397] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f: Time spent bootstrapping tablet: real 0.089s user 0.048s sys 0.031s
I20260810 22:05:15.504477 1970 raft_consensus.cc:348] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:15.504933 1972 raft_consensus.cc:348] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:15.506045 1972 raft_consensus.cc:729] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44427371e0c485eb96e803fd122ad5f, State: Initialized, Role: FOLLOWER
I20260810 22:05:15.506430 1970 raft_consensus.cc:729] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44427371e0c485eb96e803fd122ad5f, State: Initialized, Role: FOLLOWER
I20260810 22:05:15.507009 1972 consensus_queue.cc:260] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:15.507330 1970 consensus_queue.cc:260] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:15.511857 1972 ts_tablet_manager.cc:1428] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f: Time spent starting tablet: real 0.032s user 0.007s sys 0.003s
I20260810 22:05:15.512708 1970 ts_tablet_manager.cc:1428] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f: Time spent starting tablet: real 0.044s user 0.028s sys 0.012s
I20260810 22:05:15.598798 1819 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.129:32991
I20260810 22:05:15.599123 2130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.129:32991 every 8 connection(s)
I20260810 22:05:15.602032 1819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
I20260810 22:05:15.604964 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 1819
W20260810 22:05:15.606690 2132 tablet.cc:2369] T 4f0192a67e9e45a78a45868b1ce6da9d P e44427371e0c485eb96e803fd122ad5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:05:15.628018 2131 heartbeater.cc:347] Connected to a master server at 127.27.17.190:37855
I20260810 22:05:15.628531 2131 heartbeater.cc:464] Registering TS with master...
I20260810 22:05:15.629765 2131 heartbeater.cc:511] Master 127.27.17.190:37855 requested a full tablet report, sending...
I20260810 22:05:15.634770 721 ts_manager.cc:194] Re-registered known tserver with Master: e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991)
W20260810 22:05:15.637110 721 catalog_manager.cc:5408] Tablet 2e33aa436d984401993daed75b7a41b0 (table test-workload [id=122e1e087f99407c8276bb8c72ac7970]) has failed on TS e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 2e33aa436d984401993daed75b7a41b0
W20260810 22:05:15.637594 721 catalog_manager.cc:5408] Tablet 33a9afb6e00543c184b1ef3bb975cc20 (table test-workload [id=122e1e087f99407c8276bb8c72ac7970]) has failed on TS e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 33a9afb6e00543c184b1ef3bb975cc20
W20260810 22:05:15.638103 721 catalog_manager.cc:5408] Tablet 92495f335e074d6fa01e66d395101407 (table test-workload [id=122e1e087f99407c8276bb8c72ac7970]) has failed on TS e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 92495f335e074d6fa01e66d395101407
I20260810 22:05:15.641304 721 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.129:34703
I20260810 22:05:15.740551 1975 raft_consensus.cc:998] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64: Attempting to remove follower e44427371e0c485eb96e803fd122ad5f from the Raft config. Reason: The tablet replica hosted on peer e44427371e0c485eb96e803fd122ad5f has failed
I20260810 22:05:15.742795 1975 consensus_queue.cc:237] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:15.746495 1634 raft_consensus.cc:1261] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5 [term 2 FOLLOWER]: Refusing update from remote peer 912f44f41ac346ffa75962a42491ab64: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 22:05:15.747776 1848 consensus_queue.cc:1035] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 22:05:15.753580 1975 raft_consensus.cc:2932] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } }
I20260810 22:05:15.755134 1634 raft_consensus.cc:2932] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } }
I20260810 22:05:15.766724 721 catalog_manager.cc:5495] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 reported cstate change: config changed from index -1 to 2, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) evicted. New cstate: current_term: 2 leader_uuid: "912f44f41ac346ffa75962a42491ab64" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } }
I20260810 22:05:15.783525 1339 consensus_queue.cc:237] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:15.790268 1634 raft_consensus.cc:1261] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5 [term 2 FOLLOWER]: Refusing update from remote peer 912f44f41ac346ffa75962a42491ab64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260810 22:05:15.790247 1247 consensus_peers.cc:487] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 -> Peer e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Couldn't send request to peer e44427371e0c485eb96e803fd122ad5f. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 2e33aa436d984401993daed75b7a41b0. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:05:15.791713 1975 consensus_queue.cc:1035] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 22:05:15.795571 2037 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e33aa436d984401993daed75b7a41b0 with delete_type TABLET_DATA_TOMBSTONED (TS e44427371e0c485eb96e803fd122ad5f not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:52460
I20260810 22:05:15.797609 2155 ts_tablet_manager.cc:1905] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 22:05:15.798278 1848 raft_consensus.cc:2932] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } }
I20260810 22:05:15.806439 1634 raft_consensus.cc:2932] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } }
I20260810 22:05:15.811950 2131 heartbeater.cc:503] Master 127.27.17.190:37855 was elected leader, sending a full tablet report...
I20260810 22:05:15.815804 2155 ts_tablet_manager.cc:1918] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260810 22:05:15.816282 2155 log.cc:1196] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: Deleting WAL directory at /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/wal/wals/2e33aa436d984401993daed75b7a41b0
I20260810 22:05:15.818284 704 catalog_manager.cc:4841] TS e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): tablet 2e33aa436d984401993daed75b7a41b0 (table test-workload [id=122e1e087f99407c8276bb8c72ac7970]) successfully deleted
I20260810 22:05:15.826843 702 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:VOTER RPC for tablet 2e33aa436d984401993daed75b7a41b0 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20260810 22:05:15.832764 1545 consensus_peers.cc:487] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 -> Peer e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Couldn't send request to peer e44427371e0c485eb96e803fd122ad5f. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 33a9afb6e00543c184b1ef3bb975cc20. This is attempt 6: this message will repeat every 5th retry.
I20260810 22:05:15.832458 721 catalog_manager.cc:5495] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 reported cstate change: config changed from index 2 to 3, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) added. New cstate: current_term: 2 leader_uuid: "912f44f41ac346ffa75962a42491ab64" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } }
I20260810 22:05:15.836709 1832 raft_consensus.cc:998] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5: Attempting to remove follower e44427371e0c485eb96e803fd122ad5f from the Raft config. Reason: The tablet replica hosted on peer e44427371e0c485eb96e803fd122ad5f has failed
I20260810 22:05:15.838789 1832 consensus_queue.cc:237] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:15.847430 1339 raft_consensus.cc:1261] T 33a9afb6e00543c184b1ef3bb975cc20 P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Refusing update from remote peer 067bae0bd4a64f839e420a2f323862a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:05:15.848977 2160 consensus_queue.cc:1035] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 22:05:15.856411 1832 raft_consensus.cc:2932] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } }
I20260810 22:05:15.858318 1339 raft_consensus.cc:2932] T 33a9afb6e00543c184b1ef3bb975cc20 P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } }
I20260810 22:05:15.869100 720 catalog_manager.cc:5495] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 reported cstate change: config changed from index -1 to 2, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) evicted. New cstate: current_term: 1 leader_uuid: "067bae0bd4a64f839e420a2f323862a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } }
I20260810 22:05:15.881476 2037 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a9afb6e00543c184b1ef3bb975cc20 with delete_type TABLET_DATA_TOMBSTONED (TS e44427371e0c485eb96e803fd122ad5f not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:52460
I20260810 22:05:15.882874 2155 ts_tablet_manager.cc:1905] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260810 22:05:15.886209 721 catalog_manager.cc:5408] Tablet 2e33aa436d984401993daed75b7a41b0 (table test-workload [id=122e1e087f99407c8276bb8c72ac7970]) has failed on TS e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 2e33aa436d984401993daed75b7a41b0
W20260810 22:05:15.887699 721 catalog_manager.cc:5408] Tablet 92495f335e074d6fa01e66d395101407 (table test-workload [id=122e1e087f99407c8276bb8c72ac7970]) has failed on TS e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 92495f335e074d6fa01e66d395101407
I20260810 22:05:15.890064 1634 consensus_queue.cc:237] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:15.896232 1339 raft_consensus.cc:1261] T 33a9afb6e00543c184b1ef3bb975cc20 P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Refusing update from remote peer 067bae0bd4a64f839e420a2f323862a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 22:05:15.897658 2160 consensus_queue.cc:1035] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260810 22:05:15.898687 1545 consensus_peers.cc:487] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 -> Peer e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Couldn't send request to peer e44427371e0c485eb96e803fd122ad5f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:05:15.904000 2160 raft_consensus.cc:2932] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } }
I20260810 22:05:15.907310 2066 raft_consensus.cc:3037] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:05:15.910669 2155 ts_tablet_manager.cc:1918] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260810 22:05:15.911140 2155 log.cc:1196] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: Deleting WAL directory at /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/wal/wals/33a9afb6e00543c184b1ef3bb975cc20
I20260810 22:05:15.912896 704 catalog_manager.cc:4841] TS e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): tablet 33a9afb6e00543c184b1ef3bb975cc20 (table test-workload [id=122e1e087f99407c8276bb8c72ac7970]) successfully deleted
I20260810 22:05:15.915376 2066 pending_rounds.cc:76] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f: Aborting all ops after (but not including) 0
I20260810 22:05:15.914736 1339 raft_consensus.cc:2932] T 33a9afb6e00543c184b1ef3bb975cc20 P 912f44f41ac346ffa75962a42491ab64 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } }
I20260810 22:05:15.915653 2066 pending_rounds.cc:97] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260810 22:05:15.915906 2066 raft_consensus.cc:2866] T 632fa1d756704125b84b5e1772cb9076 P e44427371e0c485eb96e803fd122ad5f [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 22:05:15.920635 701 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:VOTER RPC for tablet 33a9afb6e00543c184b1ef3bb975cc20 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260810 22:05:15.927744 721 catalog_manager.cc:5495] T 33a9afb6e00543c184b1ef3bb975cc20 P 912f44f41ac346ffa75962a42491ab64 reported cstate change: config changed from index 2 to 3, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) added. New cstate: current_term: 1 leader_uuid: "067bae0bd4a64f839e420a2f323862a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } }
I20260810 22:05:16.039247 2037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 22:05:16.042035 1306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20260810 22:05:16.042871 1606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20260810 22:05:16.049984 1247 consensus_peers.cc:487] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 -> Peer e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Couldn't send request to peer e44427371e0c485eb96e803fd122ad5f. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 92495f335e074d6fa01e66d395101407. This is attempt 6: this message will repeat every 5th retry.
I20260810 22:05:16.050335 1848 raft_consensus.cc:998] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64: Attempting to remove follower e44427371e0c485eb96e803fd122ad5f from the Raft config. Reason: The tablet replica hosted on peer e44427371e0c485eb96e803fd122ad5f has failed
I20260810 22:05:16.055583 1848 consensus_queue.cc:237] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:16.072536 1634 raft_consensus.cc:1261] T 92495f335e074d6fa01e66d395101407 P 067bae0bd4a64f839e420a2f323862a5 [term 1 FOLLOWER]: Refusing update from remote peer 912f44f41ac346ffa75962a42491ab64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:05:16.081979 1975 consensus_queue.cc:1035] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 22:05:16.099927 1975 raft_consensus.cc:2932] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } }
I20260810 22:05:16.102006 1636 raft_consensus.cc:2932] T 92495f335e074d6fa01e66d395101407 P 067bae0bd4a64f839e420a2f323862a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } }
I20260810 22:05:16.114097 716 catalog_manager.cc:5495] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 reported cstate change: config changed from index -1 to 2, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) evicted. New cstate: current_term: 1 leader_uuid: "912f44f41ac346ffa75962a42491ab64" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } }
I20260810 22:05:16.135035 2037 tablet_service.cc:1514] Processing DeleteTablet for tablet 92495f335e074d6fa01e66d395101407 with delete_type TABLET_DATA_TOMBSTONED (TS e44427371e0c485eb96e803fd122ad5f not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:52460
I20260810 22:05:16.136081 2155 ts_tablet_manager.cc:1905] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 22:05:16.151292 1339 consensus_queue.cc:237] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
W20260810 22:05:16.157215 1247 consensus_peers.cc:487] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 -> Peer e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Couldn't send request to peer e44427371e0c485eb96e803fd122ad5f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:05:16.159895 1636 raft_consensus.cc:1261] T 92495f335e074d6fa01e66d395101407 P 067bae0bd4a64f839e420a2f323862a5 [term 1 FOLLOWER]: Refusing update from remote peer 912f44f41ac346ffa75962a42491ab64: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 22:05:16.161588 1975 consensus_queue.cc:1035] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 22:05:16.167229 2155 ts_tablet_manager.cc:1918] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260810 22:05:16.167641 2155 log.cc:1196] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: Deleting WAL directory at /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/wal/wals/92495f335e074d6fa01e66d395101407
I20260810 22:05:16.169739 704 catalog_manager.cc:4841] TS e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): tablet 92495f335e074d6fa01e66d395101407 (table test-workload [id=122e1e087f99407c8276bb8c72ac7970]) successfully deleted
I20260810 22:05:16.170683 1975 raft_consensus.cc:2932] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } }
I20260810 22:05:16.174945 1636 raft_consensus.cc:2932] T 92495f335e074d6fa01e66d395101407 P 067bae0bd4a64f839e420a2f323862a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } }
I20260810 22:05:16.188917 702 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:VOTER RPC for tablet 92495f335e074d6fa01e66d395101407 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260810 22:05:16.194514 716 catalog_manager.cc:5495] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64 reported cstate change: config changed from index 2 to 3, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) added. New cstate: current_term: 1 leader_uuid: "912f44f41ac346ffa75962a42491ab64" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } }
I20260810 22:05:16.313717 1975 raft_consensus.cc:998] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64: Attempting to remove follower e44427371e0c485eb96e803fd122ad5f from the Raft config. Reason: The tablet replica hosted on peer e44427371e0c485eb96e803fd122ad5f has failed
I20260810 22:05:16.315189 1975 consensus_queue.cc:237] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } }
I20260810 22:05:16.322362 2197 ts_tablet_manager.cc:927] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: Initiating tablet copy from peer 067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905)
I20260810 22:05:16.324357 2197 tablet_copy_client.cc:279] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid
I20260810 22:05:16.324296 1636 raft_consensus.cc:1261] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5 [term 2 FOLLOWER]: Refusing update from remote peer 912f44f41ac346ffa75962a42491ab64: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260810 22:05:16.324985 2197 tablet_copy_client.cc:316] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: tablet copy: Beginning tablet copy session from remote peer at address 127.27.17.131:41905
I20260810 22:05:16.325657 1975 consensus_queue.cc:1035] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 22:05:16.332620 1848 raft_consensus.cc:2932] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } }
I20260810 22:05:16.334226 1636 raft_consensus.cc:2932] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } }
I20260810 22:05:16.342398 1664 tablet_copy_service.cc:140] P 067bae0bd4a64f839e420a2f323862a5: Received BeginTabletCopySession request for tablet 33a9afb6e00543c184b1ef3bb975cc20 from peer e44427371e0c485eb96e803fd122ad5f ({username='slave'} at 127.27.17.129:60689)
I20260810 22:05:16.343050 1664 tablet_copy_service.cc:161] P 067bae0bd4a64f839e420a2f323862a5: Beginning new tablet copy session on tablet 33a9afb6e00543c184b1ef3bb975cc20 from peer e44427371e0c485eb96e803fd122ad5f at {username='slave'} at 127.27.17.129:60689: session id = e44427371e0c485eb96e803fd122ad5f-33a9afb6e00543c184b1ef3bb975cc20
I20260810 22:05:16.346818 716 catalog_manager.cc:5495] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 reported cstate change: config changed from index 3 to 4, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) evicted. New cstate: current_term: 2 leader_uuid: "912f44f41ac346ffa75962a42491ab64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } }
I20260810 22:05:16.352375 1664 tablet_copy_source_session.cc:205] T 33a9afb6e00543c184b1ef3bb975cc20 P 067bae0bd4a64f839e420a2f323862a5: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:05:16.355820 2197 ts_tablet_manager.cc:1905] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260810 22:05:16.361474 2037 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e33aa436d984401993daed75b7a41b0 with delete_type TABLET_DATA_TOMBSTONED (TS e44427371e0c485eb96e803fd122ad5f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52460
I20260810 22:05:16.363679 704 catalog_manager.cc:4841] TS e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): tablet 2e33aa436d984401993daed75b7a41b0 (table test-workload [id=122e1e087f99407c8276bb8c72ac7970]) successfully deleted
I20260810 22:05:16.365446 1339 consensus_queue.cc:237] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:16.370355 2197 ts_tablet_manager.cc:1918] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown
I20260810 22:05:16.371922 1636 raft_consensus.cc:1261] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5 [term 2 FOLLOWER]: Refusing update from remote peer 912f44f41ac346ffa75962a42491ab64: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
Master Summary
W20260810 22:05:16.372033 1247 consensus_peers.cc:487] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 -> Peer e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): Couldn't send request to peer e44427371e0c485eb96e803fd122ad5f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
UUID | Address | Status
----------------------------------+---------------------+---------
04703694c11e43c7b7c7470a0a35862b | 127.27.17.190:37855 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
I20260810 22:05:16.373412 1848 consensus_queue.cc:1035] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:37545 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
067bae0bd4a64f839e420a2f323862a5 | 127.27.17.131:41905 | HEALTHY | <none> | 1 | 0
912f44f41ac346ffa75962a42491ab64 | 127.27.17.130:39307 | HEALTHY | <none> | 4 | 0
e44427371e0c485eb96e803fd122ad5f | 127.27.17.129:32991 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.27.17.129:32991
env_inject_eio | 1 | hidden | 127.27.17.129:32991
env_inject_eio_globs | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-1/** | hidden | 127.27.17.129:32991
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.27.17.129 | experimental | 127.27.17.129:32991
local_ip_for_outbound_sockets | 127.27.17.130 | experimental | 127.27.17.130:39307
local_ip_for_outbound_sockets | 127.27.17.131 | experimental | 127.27.17.131:41905
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.27.17.129:32991
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.27.17.130:39307
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.27.17.131:41905
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:37545 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 2e33aa436d984401993daed75b7a41b0 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
912f44f41ac346ffa75962a42491ab64 (127.27.17.130:39307): RUNNING [LEADER]
067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905): RUNNING
e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 912f44f41ac346ffa75962a42491ab64
B = 067bae0bd4a64f839e420a2f323862a5
C = e44427371e0c485eb96e803fd122ad5f
I20260810 22:05:16.379315 2197 tablet_copy_client.cc:791] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: tablet copy: Starting download of 0 data blocks...
I20260810 22:05:16.380000 2197 tablet_copy_client.cc:655] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: tablet copy: Starting download of 1 WAL segments...
The consensus matrix is:
I20260810 22:05:16.379957 1975 raft_consensus.cc:2932] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } }
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 3 | Yes
B | A* B C | 2 | 3 | Yes
C | [config not available] | | |
Tablet 33a9afb6e00543c184b1ef3bb975cc20 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
I20260810 22:05:16.381620 1634 raft_consensus.cc:2932] T 2e33aa436d984401993daed75b7a41b0 P 067bae0bd4a64f839e420a2f323862a5 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } }
912f44f41ac346ffa75962a42491ab64 (127.27.17.130:39307): RUNNING
067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905): RUNNING [LEADER]
e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 912f44f41ac346ffa75962a42491ab64
B = 067bae0bd4a64f839e420a2f323862a5
C = e44427371e0c485eb96e803fd122ad5f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 3 | Yes
B | A B* C | 1 | 3 | Yes
C | [config not available] | | |
Tablet 92495f335e074d6fa01e66d395101407 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 92495f335e074d6fa01e66d395101407
912f44f41ac346ffa75962a42491ab64 (127.27.17.130:39307): RUNNING [LEADER]
067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905): RUNNING
I20260810 22:05:16.384891 2197 tablet_copy_client.cc:531] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
All reported replicas are:
A = e44427371e0c485eb96e803fd122ad5f
B = 912f44f41ac346ffa75962a42491ab64
C = 067bae0bd4a64f839e420a2f323862a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 5 | 2 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 5
First Quartile | 5
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
I20260810 22:05:16.390261 702 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:VOTER RPC for tablet 2e33aa436d984401993daed75b7a41b0 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 22:05:16.392674 716 catalog_manager.cc:5495] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64 reported cstate change: config changed from index 4 to 5, VOTER e44427371e0c485eb96e803fd122ad5f (127.27.17.129) added. New cstate: current_term: 2 leader_uuid: "912f44f41ac346ffa75962a42491ab64" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } } }
I20260810 22:05:16.394824 2197 tablet_bootstrap.cc:492] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: Bootstrap starting.
I20260810 22:05:16.402884 27718 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 22:05:16.417479 2197 tablet_bootstrap.cc:492] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:05:16.418370 2197 tablet_bootstrap.cc:492] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: Bootstrap complete.
I20260810 22:05:16.418972 2197 ts_tablet_manager.cc:1397] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: Time spent bootstrapping tablet: real 0.024s user 0.018s sys 0.004s
I20260810 22:05:16.420849 2197 raft_consensus.cc:348] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:16.421419 2197 raft_consensus.cc:729] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44427371e0c485eb96e803fd122ad5f, State: Initialized, Role: FOLLOWER
I20260810 22:05:16.422049 2197 consensus_queue.cc:260] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:16.423573 2197 ts_tablet_manager.cc:1428] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260810 22:05:16.425096 1664 tablet_copy_service.cc:342] P 067bae0bd4a64f839e420a2f323862a5: Request end of tablet copy session e44427371e0c485eb96e803fd122ad5f-33a9afb6e00543c184b1ef3bb975cc20 received from {username='slave'} at 127.27.17.129:60689
I20260810 22:05:16.425470 1664 tablet_copy_service.cc:433] P 067bae0bd4a64f839e420a2f323862a5: ending tablet copy session e44427371e0c485eb96e803fd122ad5f-33a9afb6e00543c184b1ef3bb975cc20 on tablet 33a9afb6e00543c184b1ef3bb975cc20 with peer e44427371e0c485eb96e803fd122ad5f
I20260810 22:05:16.541217 2197 ts_tablet_manager.cc:927] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: Initiating tablet copy from peer 912f44f41ac346ffa75962a42491ab64 (127.27.17.130:39307)
I20260810 22:05:16.542526 2197 tablet_copy_client.cc:279] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid
I20260810 22:05:16.543006 2197 tablet_copy_client.cc:316] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: tablet copy: Beginning tablet copy session from remote peer at address 127.27.17.130:39307
I20260810 22:05:16.553017 1361 tablet_copy_service.cc:140] P 912f44f41ac346ffa75962a42491ab64: Received BeginTabletCopySession request for tablet 92495f335e074d6fa01e66d395101407 from peer e44427371e0c485eb96e803fd122ad5f ({username='slave'} at 127.27.17.129:37113)
I20260810 22:05:16.553544 1361 tablet_copy_service.cc:161] P 912f44f41ac346ffa75962a42491ab64: Beginning new tablet copy session on tablet 92495f335e074d6fa01e66d395101407 from peer e44427371e0c485eb96e803fd122ad5f at {username='slave'} at 127.27.17.129:37113: session id = e44427371e0c485eb96e803fd122ad5f-92495f335e074d6fa01e66d395101407
I20260810 22:05:16.559988 1361 tablet_copy_source_session.cc:205] T 92495f335e074d6fa01e66d395101407 P 912f44f41ac346ffa75962a42491ab64: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:05:16.563000 2197 ts_tablet_manager.cc:1905] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260810 22:05:16.576193 2197 ts_tablet_manager.cc:1918] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown
I20260810 22:05:16.581640 2197 tablet_copy_client.cc:791] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: tablet copy: Starting download of 0 data blocks...
I20260810 22:05:16.582284 2197 tablet_copy_client.cc:655] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: tablet copy: Starting download of 1 WAL segments...
I20260810 22:05:16.586073 2197 tablet_copy_client.cc:531] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:05:16.591895 2197 tablet_bootstrap.cc:492] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: Bootstrap starting.
I20260810 22:05:16.613272 2197 tablet_bootstrap.cc:492] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:05:16.615077 2197 tablet_bootstrap.cc:492] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: Bootstrap complete.
I20260810 22:05:16.615784 2197 ts_tablet_manager.cc:1397] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.004s
I20260810 22:05:16.618479 2197 raft_consensus.cc:348] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:16.619184 2197 raft_consensus.cc:729] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44427371e0c485eb96e803fd122ad5f, State: Initialized, Role: FOLLOWER
I20260810 22:05:16.619851 2197 consensus_queue.cc:260] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:16.622069 2197 ts_tablet_manager.cc:1428] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260810 22:05:16.623813 1361 tablet_copy_service.cc:342] P 912f44f41ac346ffa75962a42491ab64: Request end of tablet copy session e44427371e0c485eb96e803fd122ad5f-92495f335e074d6fa01e66d395101407 received from {username='slave'} at 127.27.17.129:37113
I20260810 22:05:16.624220 1361 tablet_copy_service.cc:433] P 912f44f41ac346ffa75962a42491ab64: ending tablet copy session e44427371e0c485eb96e803fd122ad5f-92495f335e074d6fa01e66d395101407 on tablet 92495f335e074d6fa01e66d395101407 with peer e44427371e0c485eb96e803fd122ad5f
E20260810 22:05:16.669266 2124 env_posix.cc:432] I/O error, context: INJECTED FAILURE
I20260810 22:05:16.766343 2197 ts_tablet_manager.cc:927] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: Initiating tablet copy from peer 912f44f41ac346ffa75962a42491ab64 (127.27.17.130:39307)
I20260810 22:05:16.767501 2197 tablet_copy_client.cc:279] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid
I20260810 22:05:16.767956 2197 tablet_copy_client.cc:316] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: tablet copy: Beginning tablet copy session from remote peer at address 127.27.17.130:39307
I20260810 22:05:16.769275 1361 tablet_copy_service.cc:140] P 912f44f41ac346ffa75962a42491ab64: Received BeginTabletCopySession request for tablet 2e33aa436d984401993daed75b7a41b0 from peer e44427371e0c485eb96e803fd122ad5f ({username='slave'} at 127.27.17.129:37113)
I20260810 22:05:16.769831 1361 tablet_copy_service.cc:161] P 912f44f41ac346ffa75962a42491ab64: Beginning new tablet copy session on tablet 2e33aa436d984401993daed75b7a41b0 from peer e44427371e0c485eb96e803fd122ad5f at {username='slave'} at 127.27.17.129:37113: session id = e44427371e0c485eb96e803fd122ad5f-2e33aa436d984401993daed75b7a41b0
I20260810 22:05:16.776310 1361 tablet_copy_source_session.cc:205] T 2e33aa436d984401993daed75b7a41b0 P 912f44f41ac346ffa75962a42491ab64: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:05:16.778321 2197 ts_tablet_manager.cc:1905] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260810 22:05:16.789948 2197 ts_tablet_manager.cc:1918] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown
I20260810 22:05:16.806150 2197 tablet_copy_client.cc:791] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: tablet copy: Starting download of 0 data blocks...
I20260810 22:05:16.806761 2197 tablet_copy_client.cc:655] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: tablet copy: Starting download of 1 WAL segments...
I20260810 22:05:16.810861 2197 tablet_copy_client.cc:531] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:05:16.816354 1606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 22:05:16.819305 2197 tablet_bootstrap.cc:492] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: Bootstrap starting.
I20260810 22:05:16.831164 2037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 22:05:16.831431 1306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20260810 22:05:16.847616 2197 tablet_bootstrap.cc:492] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:05:16.848897 2197 tablet_bootstrap.cc:492] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: Bootstrap complete.
I20260810 22:05:16.849618 2197 ts_tablet_manager.cc:1397] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: Time spent bootstrapping tablet: real 0.030s user 0.030s sys 0.000s
I20260810 22:05:16.851819 2197 raft_consensus.cc:348] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:16.852499 2197 raft_consensus.cc:729] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44427371e0c485eb96e803fd122ad5f, State: Initialized, Role: FOLLOWER
I20260810 22:05:16.853232 2197 consensus_queue.cc:260] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "912f44f41ac346ffa75962a42491ab64" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 39307 } } peers { permanent_uuid: "067bae0bd4a64f839e420a2f323862a5" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 41905 } } peers { permanent_uuid: "e44427371e0c485eb96e803fd122ad5f" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 32991 } }
I20260810 22:05:16.855309 2197 ts_tablet_manager.cc:1428] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:05:16.857800 1361 tablet_copy_service.cc:342] P 912f44f41ac346ffa75962a42491ab64: Request end of tablet copy session e44427371e0c485eb96e803fd122ad5f-2e33aa436d984401993daed75b7a41b0 received from {username='slave'} at 127.27.17.129:37113
I20260810 22:05:16.858171 1361 tablet_copy_service.cc:433] P 912f44f41ac346ffa75962a42491ab64: ending tablet copy session e44427371e0c485eb96e803fd122ad5f-2e33aa436d984401993daed75b7a41b0 on tablet 2e33aa436d984401993daed75b7a41b0 with peer e44427371e0c485eb96e803fd122ad5f
I20260810 22:05:16.892802 2066 raft_consensus.cc:1203] T 33a9afb6e00543c184b1ef3bb975cc20 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20260810 22:05:17.056167 2066 raft_consensus.cc:1203] T 92495f335e074d6fa01e66d395101407 P e44427371e0c485eb96e803fd122ad5f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
04703694c11e43c7b7c7470a0a35862b | 127.27.17.190:37855 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:37545 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
067bae0bd4a64f839e420a2f323862a5 | 127.27.17.131:41905 | HEALTHY | <none> | 1 | 0
912f44f41ac346ffa75962a42491ab64 | 127.27.17.130:39307 | HEALTHY | <none> | 4 | 0
e44427371e0c485eb96e803fd122ad5f | 127.27.17.129:32991 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.27.17.129:32991
env_inject_eio | 1 | hidden | 127.27.17.129:32991
env_inject_eio_globs | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-1/** | hidden | 127.27.17.129:32991
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.27.17.129 | experimental | 127.27.17.129:32991
local_ip_for_outbound_sockets | 127.27.17.130 | experimental | 127.27.17.130:39307
local_ip_for_outbound_sockets | 127.27.17.131 | experimental | 127.27.17.131:41905
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.27.17.129:32991
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.27.17.130:39307
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.27.17.131:41905
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:37545 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 2e33aa436d984401993daed75b7a41b0 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
912f44f41ac346ffa75962a42491ab64 (127.27.17.130:39307): RUNNING [LEADER]
067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905): RUNNING
e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): not running
State: BOOTSTRAPPING
Data state: TABLET_DATA_READY
Last status: Bootstrap starting.
All reported replicas are:
A = 912f44f41ac346ffa75962a42491ab64
B = 067bae0bd4a64f839e420a2f323862a5
C = e44427371e0c485eb96e803fd122ad5f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 5 | Yes
B | A* B C | 2 | 5 | Yes
C | A B C | 2 | 5 | Yes
Tablet 33a9afb6e00543c184b1ef3bb975cc20 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
912f44f41ac346ffa75962a42491ab64 (127.27.17.130:39307): RUNNING
067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905): RUNNING [LEADER]
e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): RUNNING
All reported replicas are:
A = 912f44f41ac346ffa75962a42491ab64
B = 067bae0bd4a64f839e420a2f323862a5
C = e44427371e0c485eb96e803fd122ad5f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 3 | Yes
B | A B* C | 1 | 3 | Yes
C | A B C | 1 | 3 | Yes
Tablet 92495f335e074d6fa01e66d395101407 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
912f44f41ac346ffa75962a42491ab64 (127.27.17.130:39307): RUNNING [LEADER]
067bae0bd4a64f839e420a2f323862a5 (127.27.17.131:41905): RUNNING
e44427371e0c485eb96e803fd122ad5f (127.27.17.129:32991): RUNNING
All reported replicas are:
A = 912f44f41ac346ffa75962a42491ab64
B = 067bae0bd4a64f839e420a2f323862a5
C = e44427371e0c485eb96e803fd122ad5f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 3 | Yes
B | A* B C | 1 | 3 | Yes
C | A B C | 1 | 3 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 5 | 2 | 0 | 1 | 2
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 5
First Quartile | 5
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 22:05:17.083608 27718 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 22:05:17.319983 2066 raft_consensus.cc:1203] T 2e33aa436d984401993daed75b7a41b0 P e44427371e0c485eb96e803fd122ad5f [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.4->[2.5-2.5] Dedup: 2.5->[]
I20260810 22:05:17.563596 1606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 22:05:17.567832 1306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20260810 22:05:17.569469 2037 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
04703694c11e43c7b7c7470a0a35862b | 127.27.17.190:37855 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:37545 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
067bae0bd4a64f839e420a2f323862a5 | 127.27.17.131:41905 | HEALTHY | <none> | 1 | 0
912f44f41ac346ffa75962a42491ab64 | 127.27.17.130:39307 | HEALTHY | <none> | 4 | 0
e44427371e0c485eb96e803fd122ad5f | 127.27.17.129:32991 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.27.17.129:32991
env_inject_eio | 1 | hidden | 127.27.17.129:32991
env_inject_eio_globs | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-1/** | hidden | 127.27.17.129:32991
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.27.17.129 | experimental | 127.27.17.129:32991
local_ip_for_outbound_sockets | 127.27.17.130 | experimental | 127.27.17.130:39307
local_ip_for_outbound_sockets | 127.27.17.131 | experimental | 127.27.17.131:41905
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.27.17.129:32991
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.27.17.130:39307
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.27.17.131:41905
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:37545 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 5 | 5 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 5
First Quartile | 5
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260810 22:05:17.786350 27718 log_verifier.cc:126] Checking tablet 2e33aa436d984401993daed75b7a41b0
I20260810 22:05:17.853602 27718 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 2e33aa436d984401993daed75b7a41b0
I20260810 22:05:17.853907 27718 log_verifier.cc:126] Checking tablet 33a9afb6e00543c184b1ef3bb975cc20
I20260810 22:05:17.924489 27718 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 33a9afb6e00543c184b1ef3bb975cc20
I20260810 22:05:17.924764 27718 log_verifier.cc:126] Checking tablet 4f0192a67e9e45a78a45868b1ce6da9d
I20260810 22:05:17.989900 27718 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 4f0192a67e9e45a78a45868b1ce6da9d
I20260810 22:05:17.990154 27718 log_verifier.cc:126] Checking tablet 632fa1d756704125b84b5e1772cb9076
I20260810 22:05:18.056138 27718 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 632fa1d756704125b84b5e1772cb9076
I20260810 22:05:18.056393 27718 log_verifier.cc:126] Checking tablet 92495f335e074d6fa01e66d395101407
I20260810 22:05:18.123581 27718 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 92495f335e074d6fa01e66d395101407
I20260810 22:05:18.488597 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 1819
I20260810 22:05:18.546957 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 1102
I20260810 22:05:18.602133 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 1402
I20260810 22:05:18.658721 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 559
2026-08-10T22:05:18Z chronyd exiting
[ OK ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/4 (16085 ms)
[ RUN ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/5
2026-08-10T22:05:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:05:18Z Disabled control of system clock
I20260810 22:05:18.798923 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.190:33447
--webserver_interface=127.27.17.190
--webserver_port=0
--builtin_ntp_servers=127.27.17.148:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.190:33447
--raft_prepare_replacement_before_eviction=true with env {}
W20260810 22:05:19.122241 2308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:05:19.122905 2308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:05:19.123392 2308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:05:19.156762 2308 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20260810 22:05:19.157222 2308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:05:19.157567 2308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:05:19.157838 2308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:05:19.158100 2308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:05:19.194563 2308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.190:33447
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.190:33447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
--webserver_interface=127.27.17.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:05:19.195948 2308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:05:19.197676 2308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:05:19.214865 2318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:19.215721 2321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:19.217123 2308 server_base.cc:1034] running on GCE node
W20260810 22:05:19.216918 2323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:20.415236 2308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:05:20.418041 2308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:05:20.419394 2308 hybrid_clock.cc:648] HybridClock initialized: now 1786399520419349 us; error 68 us; skew 500 ppm
I20260810 22:05:20.420264 2308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:05:20.434973 2308 webserver.cc:460] Webserver started at http://127.27.17.190:34083/ using document root <none> and password file <none>
I20260810 22:05:20.436712 2308 fs_manager.cc:359] Metadata directory not provided
I20260810 22:05:20.436975 2308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:05:20.437821 2308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:05:20.446943 2308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-0/instance:
uuid: "12ce8059216a4cb68b7ba53cac315b54"
format_stamp: "Formatted at 2026-08-10 22:05:20 on dist-test-slave-bw88"
I20260810 22:05:20.448174 2308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-1/instance:
uuid: "12ce8059216a4cb68b7ba53cac315b54"
format_stamp: "Formatted at 2026-08-10 22:05:20 on dist-test-slave-bw88"
I20260810 22:05:20.449093 2308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-2/instance:
uuid: "12ce8059216a4cb68b7ba53cac315b54"
format_stamp: "Formatted at 2026-08-10 22:05:20 on dist-test-slave-bw88"
I20260810 22:05:20.450013 2308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-3/instance:
uuid: "12ce8059216a4cb68b7ba53cac315b54"
format_stamp: "Formatted at 2026-08-10 22:05:20 on dist-test-slave-bw88"
I20260810 22:05:20.450903 2308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-4/instance:
uuid: "12ce8059216a4cb68b7ba53cac315b54"
format_stamp: "Formatted at 2026-08-10 22:05:20 on dist-test-slave-bw88"
I20260810 22:05:20.451820 2308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/wal/instance:
uuid: "12ce8059216a4cb68b7ba53cac315b54"
format_stamp: "Formatted at 2026-08-10 22:05:20 on dist-test-slave-bw88"
I20260810 22:05:20.475550 2308 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.018s sys 0.006s
I20260810 22:05:20.942237 2308 fs_manager.cc:725] Time spent opening block manager: real 0.462s user 0.003s sys 0.004s
I20260810 22:05:20.942628 2308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/wal
uuid: "12ce8059216a4cb68b7ba53cac315b54"
format_stamp: "Formatted at 2026-08-10 22:05:20 on dist-test-slave-bw88"
I20260810 22:05:20.943060 2308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:05:20.983230 2308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:05:20.984849 2308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:05:20.985353 2308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:05:21.041198 2308 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.190:33447
I20260810 22:05:21.041505 2531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.190:33447 every 8 connection(s)
I20260810 22:05:21.044016 2308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
I20260810 22:05:21.052760 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 2308
I20260810 22:05:21.053576 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/wal/instance
I20260810 22:05:21.074362 2532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54: Bootstrap starting.
I20260810 22:05:21.081386 2532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:21.083606 2532 log.cc:824] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54: Log is configured to *not* fsync() on all Append() calls
I20260810 22:05:21.102774 2532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54: No bootstrap required, opened a new log
I20260810 22:05:21.123509 2532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ce8059216a4cb68b7ba53cac315b54" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 33447 } }
I20260810 22:05:21.124215 2532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:21.124421 2532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ce8059216a4cb68b7ba53cac315b54, State: Initialized, Role: FOLLOWER
I20260810 22:05:21.125053 2532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ce8059216a4cb68b7ba53cac315b54" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 33447 } }
I20260810 22:05:21.125612 2532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:05:21.125842 2532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:05:21.126111 2532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:21.129992 2532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ce8059216a4cb68b7ba53cac315b54" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 33447 } }
I20260810 22:05:21.130668 2532 leader_election.cc:302] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12ce8059216a4cb68b7ba53cac315b54; no voters:
I20260810 22:05:21.132622 2532 leader_election.cc:288] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:05:21.133059 2541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:21.136526 2541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54 [term 1 LEADER]: Becoming Leader. State: Replica: 12ce8059216a4cb68b7ba53cac315b54, State: Running, Role: LEADER
I20260810 22:05:21.137377 2541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ce8059216a4cb68b7ba53cac315b54" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 33447 } }
I20260810 22:05:21.138373 2532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:05:21.150800 2543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12ce8059216a4cb68b7ba53cac315b54. Latest consensus state: current_term: 1 leader_uuid: "12ce8059216a4cb68b7ba53cac315b54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ce8059216a4cb68b7ba53cac315b54" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 33447 } } }
I20260810 22:05:21.147802 2542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12ce8059216a4cb68b7ba53cac315b54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ce8059216a4cb68b7ba53cac315b54" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 33447 } } }
I20260810 22:05:21.152442 2543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54 [sys.catalog]: This master's current role is: LEADER
I20260810 22:05:21.152724 2542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54 [sys.catalog]: This master's current role is: LEADER
I20260810 22:05:21.156800 2548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:05:21.170363 2548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:05:21.186760 2548 catalog_manager.cc:1347] Generated new cluster ID: ed7c3aeaf78e437d8e74664d5eadc804
I20260810 22:05:21.187070 2548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:05:21.213505 2548 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:05:21.216192 2548 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:05:21.242043 2548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12ce8059216a4cb68b7ba53cac315b54: Generated new TSK 0
I20260810 22:05:21.243052 2548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:05:21.263157 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.129:0
--local_ip_for_outbound_sockets=127.27.17.129
--webserver_interface=127.27.17.129
--webserver_port=0
--tserver_master_addrs=127.27.17.190:33447
--builtin_ntp_servers=127.27.17.148:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20260810 22:05:21.597522 2563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:05:21.598100 2563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:05:21.598613 2563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:05:21.631153 2563 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20260810 22:05:21.631592 2563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:05:21.632427 2563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.129
I20260810 22:05:21.669195 2563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.27.17.129
--webserver_port=0
--tserver_master_addrs=127.27.17.190:33447
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.129
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:05:21.670543 2563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:05:21.672290 2563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:05:21.694214 2574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:22.803697 2528 debug-util.cc:398] Leaking SignalData structure 0x7b080007e060 after lost signal to thread 2310
W20260810 22:05:22.804515 2528 debug-util.cc:398] Leaking SignalData structure 0x7b080007e800 after lost signal to thread 2531
W20260810 22:05:21.695444 2573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:21.696048 2563 server_base.cc:1034] running on GCE node
W20260810 22:05:21.695218 2580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:22.976529 2563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:05:22.979707 2563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:05:22.981290 2563 hybrid_clock.cc:648] HybridClock initialized: now 1786399522981239 us; error 44 us; skew 500 ppm
I20260810 22:05:22.982524 2563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:05:22.991101 2563 webserver.cc:460] Webserver started at http://127.27.17.129:35231/ using document root <none> and password file <none>
I20260810 22:05:22.993505 2563 fs_manager.cc:359] Metadata directory not provided
I20260810 22:05:22.993813 2563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:05:22.994756 2563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:05:23.003032 2563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-0/instance:
uuid: "742853e1f3ae4ac1b86265b6adb193f9"
format_stamp: "Formatted at 2026-08-10 22:05:22 on dist-test-slave-bw88"
I20260810 22:05:23.004736 2563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-1/instance:
uuid: "742853e1f3ae4ac1b86265b6adb193f9"
format_stamp: "Formatted at 2026-08-10 22:05:22 on dist-test-slave-bw88"
I20260810 22:05:23.006078 2563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-2/instance:
uuid: "742853e1f3ae4ac1b86265b6adb193f9"
format_stamp: "Formatted at 2026-08-10 22:05:22 on dist-test-slave-bw88"
I20260810 22:05:23.007022 2563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-3/instance:
uuid: "742853e1f3ae4ac1b86265b6adb193f9"
format_stamp: "Formatted at 2026-08-10 22:05:22 on dist-test-slave-bw88"
I20260810 22:05:23.007936 2563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-4/instance:
uuid: "742853e1f3ae4ac1b86265b6adb193f9"
format_stamp: "Formatted at 2026-08-10 22:05:22 on dist-test-slave-bw88"
I20260810 22:05:23.008837 2563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/wal/instance:
uuid: "742853e1f3ae4ac1b86265b6adb193f9"
format_stamp: "Formatted at 2026-08-10 22:05:22 on dist-test-slave-bw88"
I20260810 22:05:23.034000 2563 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.015s sys 0.008s
I20260810 22:05:23.591670 2563 fs_manager.cc:725] Time spent opening block manager: real 0.553s user 0.005s sys 0.000s
I20260810 22:05:23.592060 2563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/wal
uuid: "742853e1f3ae4ac1b86265b6adb193f9"
format_stamp: "Formatted at 2026-08-10 22:05:22 on dist-test-slave-bw88"
I20260810 22:05:23.592460 2563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:05:23.631155 2563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:05:23.632633 2563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:05:23.633095 2563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:05:23.635529 2563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:05:23.639885 2563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:05:23.640105 2563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:23.640372 2563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:05:23.640513 2563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:23.803025 2563 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.129:33615
I20260810 22:05:23.803351 2858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.129:33615 every 8 connection(s)
I20260810 22:05:23.805698 2563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
I20260810 22:05:23.810189 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 2563
I20260810 22:05:23.810768 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/wal/instance
I20260810 22:05:23.820916 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.130:0
--local_ip_for_outbound_sockets=127.27.17.130
--webserver_interface=127.27.17.130
--webserver_port=0
--tserver_master_addrs=127.27.17.190:33447
--builtin_ntp_servers=127.27.17.148:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260810 22:05:23.859838 2859 heartbeater.cc:347] Connected to a master server at 127.27.17.190:33447
I20260810 22:05:23.860381 2859 heartbeater.cc:464] Registering TS with master...
I20260810 22:05:23.861729 2859 heartbeater.cc:511] Master 127.27.17.190:33447 requested a full tablet report, sending...
I20260810 22:05:23.865159 2475 ts_manager.cc:194] Registered new tserver with Master: 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615)
I20260810 22:05:23.868469 2475 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.129:55475
W20260810 22:05:24.148800 2863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:05:24.149355 2863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:05:24.149940 2863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:05:24.184667 2863 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20260810 22:05:24.185114 2863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:05:24.185968 2863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.130
I20260810 22:05:24.220479 2863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
--webserver_interface=127.27.17.130
--webserver_port=0
--tserver_master_addrs=127.27.17.190:33447
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.130
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:05:24.221879 2863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:05:24.223687 2863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:05:24.246999 2871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:24.873644 2859 heartbeater.cc:503] Master 127.27.17.190:33447 was elected leader, sending a full tablet report...
W20260810 22:05:24.247720 2870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:24.249162 2863 server_base.cc:1034] running on GCE node
W20260810 22:05:24.249069 2877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:25.515136 2863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:05:25.518085 2863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:05:25.519531 2863 hybrid_clock.cc:648] HybridClock initialized: now 1786399525519495 us; error 57 us; skew 500 ppm
I20260810 22:05:25.520305 2863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:05:25.532300 2863 webserver.cc:460] Webserver started at http://127.27.17.130:34899/ using document root <none> and password file <none>
I20260810 22:05:25.533901 2863 fs_manager.cc:359] Metadata directory not provided
I20260810 22:05:25.534111 2863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:05:25.534754 2863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:05:25.540086 2863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-0/instance:
uuid: "53b64278bdc549e9a56784b6b3295490"
format_stamp: "Formatted at 2026-08-10 22:05:25 on dist-test-slave-bw88"
I20260810 22:05:25.541327 2863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-1/instance:
uuid: "53b64278bdc549e9a56784b6b3295490"
format_stamp: "Formatted at 2026-08-10 22:05:25 on dist-test-slave-bw88"
I20260810 22:05:25.542438 2863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-2/instance:
uuid: "53b64278bdc549e9a56784b6b3295490"
format_stamp: "Formatted at 2026-08-10 22:05:25 on dist-test-slave-bw88"
I20260810 22:05:25.543280 2863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-3/instance:
uuid: "53b64278bdc549e9a56784b6b3295490"
format_stamp: "Formatted at 2026-08-10 22:05:25 on dist-test-slave-bw88"
I20260810 22:05:25.544091 2863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-4/instance:
uuid: "53b64278bdc549e9a56784b6b3295490"
format_stamp: "Formatted at 2026-08-10 22:05:25 on dist-test-slave-bw88"
I20260810 22:05:25.544893 2863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/wal/instance:
uuid: "53b64278bdc549e9a56784b6b3295490"
format_stamp: "Formatted at 2026-08-10 22:05:25 on dist-test-slave-bw88"
I20260810 22:05:25.567920 2863 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.015s sys 0.007s
I20260810 22:05:25.996551 2863 fs_manager.cc:725] Time spent opening block manager: real 0.424s user 0.006s sys 0.001s
I20260810 22:05:25.996917 2863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/wal
uuid: "53b64278bdc549e9a56784b6b3295490"
format_stamp: "Formatted at 2026-08-10 22:05:25 on dist-test-slave-bw88"
I20260810 22:05:25.997290 2863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:05:26.032804 2863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:05:26.034322 2863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:05:26.034735 2863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:05:26.036753 2863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:05:26.040726 2863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:05:26.040915 2863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:26.041201 2863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:05:26.041390 2863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260810 22:05:26.159657 2863 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.130:44557
I20260810 22:05:26.159926 3158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.130:44557 every 8 connection(s)
I20260810 22:05:26.162384 2863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
I20260810 22:05:26.169515 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 2863
I20260810 22:05:26.170111 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/wal/instance
I20260810 22:05:26.176877 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.131:0
--local_ip_for_outbound_sockets=127.27.17.131
--webserver_interface=127.27.17.131
--webserver_port=0
--tserver_master_addrs=127.27.17.190:33447
--builtin_ntp_servers=127.27.17.148:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260810 22:05:26.188287 3159 heartbeater.cc:347] Connected to a master server at 127.27.17.190:33447
I20260810 22:05:26.188774 3159 heartbeater.cc:464] Registering TS with master...
I20260810 22:05:26.189926 3159 heartbeater.cc:511] Master 127.27.17.190:33447 requested a full tablet report, sending...
I20260810 22:05:26.192476 2475 ts_manager.cc:194] Registered new tserver with Master: 53b64278bdc549e9a56784b6b3295490 (127.27.17.130:44557)
I20260810 22:05:26.194919 2475 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.130:33977
W20260810 22:05:26.495239 3164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:05:26.495724 3164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:05:26.496162 3164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:05:26.527876 3164 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20260810 22:05:26.528224 3164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:05:26.528952 3164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.131
I20260810 22:05:26.564680 3164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
--webserver_interface=127.27.17.131
--webserver_port=0
--tserver_master_addrs=127.27.17.190:33447
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.131
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:05:26.565980 3164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:05:26.567560 3164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:05:26.586243 3170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:27.199163 3159 heartbeater.cc:503] Master 127.27.17.190:33447 was elected leader, sending a full tablet report...
I20260810 22:05:26.587570 3164 server_base.cc:1034] running on GCE node
W20260810 22:05:26.586342 3174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:26.586760 3171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:27.797796 3164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:05:27.800742 3164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:05:27.802191 3164 hybrid_clock.cc:648] HybridClock initialized: now 1786399527802171 us; error 44 us; skew 500 ppm
I20260810 22:05:27.803014 3164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:05:27.810178 3164 webserver.cc:460] Webserver started at http://127.27.17.131:36479/ using document root <none> and password file <none>
I20260810 22:05:27.811681 3164 fs_manager.cc:359] Metadata directory not provided
I20260810 22:05:27.811940 3164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:05:27.812631 3164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:05:27.817759 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-0/instance:
uuid: "bdb9b70cb73a412ab0d4049695126b0a"
format_stamp: "Formatted at 2026-08-10 22:05:27 on dist-test-slave-bw88"
I20260810 22:05:27.818861 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-1/instance:
uuid: "bdb9b70cb73a412ab0d4049695126b0a"
format_stamp: "Formatted at 2026-08-10 22:05:27 on dist-test-slave-bw88"
I20260810 22:05:27.819697 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-2/instance:
uuid: "bdb9b70cb73a412ab0d4049695126b0a"
format_stamp: "Formatted at 2026-08-10 22:05:27 on dist-test-slave-bw88"
I20260810 22:05:27.820600 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-3/instance:
uuid: "bdb9b70cb73a412ab0d4049695126b0a"
format_stamp: "Formatted at 2026-08-10 22:05:27 on dist-test-slave-bw88"
I20260810 22:05:27.821507 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-4/instance:
uuid: "bdb9b70cb73a412ab0d4049695126b0a"
format_stamp: "Formatted at 2026-08-10 22:05:27 on dist-test-slave-bw88"
I20260810 22:05:27.822335 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/wal/instance:
uuid: "bdb9b70cb73a412ab0d4049695126b0a"
format_stamp: "Formatted at 2026-08-10 22:05:27 on dist-test-slave-bw88"
I20260810 22:05:27.845582 3164 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.019s sys 0.004s
I20260810 22:05:28.309443 3164 fs_manager.cc:725] Time spent opening block manager: real 0.459s user 0.004s sys 0.000s
I20260810 22:05:28.309816 3164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/wal
uuid: "bdb9b70cb73a412ab0d4049695126b0a"
format_stamp: "Formatted at 2026-08-10 22:05:27 on dist-test-slave-bw88"
I20260810 22:05:28.310231 3164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:05:28.346253 3164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:05:28.347703 3164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:05:28.348158 3164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:05:28.350194 3164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:05:28.354033 3164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:05:28.354274 3164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:28.354548 3164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:05:28.354719 3164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:28.481976 3164 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.131:45861
I20260810 22:05:28.482183 3447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.131:45861 every 8 connection(s)
I20260810 22:05:28.484778 3164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
I20260810 22:05:28.495298 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 3164
I20260810 22:05:28.495769 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/wal/instance
I20260810 22:05:28.502785 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.132:0
--local_ip_for_outbound_sockets=127.27.17.132
--webserver_interface=127.27.17.132
--webserver_port=0
--tserver_master_addrs=127.27.17.190:33447
--builtin_ntp_servers=127.27.17.148:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260810 22:05:28.510232 3450 heartbeater.cc:347] Connected to a master server at 127.27.17.190:33447
I20260810 22:05:28.510639 3450 heartbeater.cc:464] Registering TS with master...
I20260810 22:05:28.511612 3450 heartbeater.cc:511] Master 127.27.17.190:33447 requested a full tablet report, sending...
I20260810 22:05:28.514207 2475 ts_manager.cc:194] Registered new tserver with Master: bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131:45861)
I20260810 22:05:28.515938 2475 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.131:35255
W20260810 22:05:28.817838 3455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:05:28.818324 3455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:05:28.818936 3455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:05:28.849620 3455 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20260810 22:05:28.850046 3455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:05:28.850845 3455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.132
I20260810 22:05:28.885886 3455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-0/info.pb
--webserver_interface=127.27.17.132
--webserver_port=0
--tserver_master_addrs=127.27.17.190:33447
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.132
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:05:28.887184 3455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:05:28.888933 3455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:05:28.909096 3465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:29.519867 3450 heartbeater.cc:503] Master 127.27.17.190:33447 was elected leader, sending a full tablet report...
W20260810 22:05:28.909358 3464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:28.909737 3467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:30.092129 3466 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 22:05:30.092274 3455 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 22:05:30.096642 3455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:05:30.099066 3455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:05:30.100481 3455 hybrid_clock.cc:648] HybridClock initialized: now 1786399530100447 us; error 73 us; skew 500 ppm
I20260810 22:05:30.101429 3455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:05:30.108691 3455 webserver.cc:460] Webserver started at http://127.27.17.132:37285/ using document root <none> and password file <none>
I20260810 22:05:30.110239 3455 fs_manager.cc:359] Metadata directory not provided
I20260810 22:05:30.110496 3455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:05:30.111153 3455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:05:30.116182 3455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-0/instance:
uuid: "da5bd62264364fd198587af606253dd1"
format_stamp: "Formatted at 2026-08-10 22:05:30 on dist-test-slave-bw88"
I20260810 22:05:30.117470 3455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-1/instance:
uuid: "da5bd62264364fd198587af606253dd1"
format_stamp: "Formatted at 2026-08-10 22:05:30 on dist-test-slave-bw88"
I20260810 22:05:30.118587 3455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-2/instance:
uuid: "da5bd62264364fd198587af606253dd1"
format_stamp: "Formatted at 2026-08-10 22:05:30 on dist-test-slave-bw88"
I20260810 22:05:30.119493 3455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-3/instance:
uuid: "da5bd62264364fd198587af606253dd1"
format_stamp: "Formatted at 2026-08-10 22:05:30 on dist-test-slave-bw88"
I20260810 22:05:30.120333 3455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-4/instance:
uuid: "da5bd62264364fd198587af606253dd1"
format_stamp: "Formatted at 2026-08-10 22:05:30 on dist-test-slave-bw88"
I20260810 22:05:30.121150 3455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/wal/instance:
uuid: "da5bd62264364fd198587af606253dd1"
format_stamp: "Formatted at 2026-08-10 22:05:30 on dist-test-slave-bw88"
I20260810 22:05:30.145535 3455 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.016s sys 0.008s
I20260810 22:05:30.623528 3455 fs_manager.cc:725] Time spent opening block manager: real 0.473s user 0.005s sys 0.001s
I20260810 22:05:30.624020 3455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/wal
uuid: "da5bd62264364fd198587af606253dd1"
format_stamp: "Formatted at 2026-08-10 22:05:30 on dist-test-slave-bw88"
I20260810 22:05:30.624648 3455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/wal
5 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:05:30.660403 3455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:05:30.662032 3455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:05:30.662613 3455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:05:30.665001 3455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:05:30.669159 3455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:05:30.669440 3455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260810 22:05:30.669763 3455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:05:30.669910 3455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:30.791682 3455 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.132:41969
I20260810 22:05:30.791937 3747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.132:41969 every 8 connection(s)
I20260810 22:05:30.794420 3455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-0/info.pb
I20260810 22:05:30.800462 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 3455
I20260810 22:05:30.801767 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/wal/instance
I20260810 22:05:30.830231 3748 heartbeater.cc:347] Connected to a master server at 127.27.17.190:33447
I20260810 22:05:30.830672 3748 heartbeater.cc:464] Registering TS with master...
I20260810 22:05:30.831663 3748 heartbeater.cc:511] Master 127.27.17.190:33447 requested a full tablet report, sending...
I20260810 22:05:30.834139 2475 ts_manager.cc:194] Registered new tserver with Master: da5bd62264364fd198587af606253dd1 (127.27.17.132:41969)
I20260810 22:05:30.836236 2475 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.132:39383
I20260810 22:05:30.841641 27718 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20260810 22:05:30.878886 27718 test_util.cc:268] Using random seed: -1921579119
I20260810 22:05:30.930313 2475 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51070:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\231\231\231\031\004\001\0002333\004\001\000\313\314\314L\004\001\000dfff"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 22:05:31.026486 3068 tablet_service.cc:1467] Processing CreateTablet for tablet 7230e7ad4d70476e85b798e58579d61f (DEFAULT_TABLE table=test-workload [id=3cc1488e248b4e99a0705e301f54d797]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260810 22:05:31.025895 3655 tablet_service.cc:1467] Processing CreateTablet for tablet 7230e7ad4d70476e85b798e58579d61f (DEFAULT_TABLE table=test-workload [id=3cc1488e248b4e99a0705e301f54d797]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260810 22:05:31.027026 2772 tablet_service.cc:1467] Processing CreateTablet for tablet 391eba1c39404071969e21ddd5abef69 (DEFAULT_TABLE table=test-workload [id=3cc1488e248b4e99a0705e301f54d797]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260810 22:05:31.028637 2769 tablet_service.cc:1467] Processing CreateTablet for tablet 943ec2cbc29e44e3bcbe0e1e5b039f55 (DEFAULT_TABLE table=test-workload [id=3cc1488e248b4e99a0705e301f54d797]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260810 22:05:31.029493 2768 tablet_service.cc:1467] Processing CreateTablet for tablet 159f82c64cc8402fb2570de1b5223e78 (DEFAULT_TABLE table=test-workload [id=3cc1488e248b4e99a0705e301f54d797]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260810 22:05:31.030457 3069 tablet_service.cc:1467] Processing CreateTablet for tablet cfb52501fc724335a0c3d3e6a7ee95db (DEFAULT_TABLE table=test-workload [id=3cc1488e248b4e99a0705e301f54d797]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260810 22:05:31.031574 2770 tablet_service.cc:1467] Processing CreateTablet for tablet cfb52501fc724335a0c3d3e6a7ee95db (DEFAULT_TABLE table=test-workload [id=3cc1488e248b4e99a0705e301f54d797]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260810 22:05:31.022349 3071 tablet_service.cc:1467] Processing CreateTablet for tablet 391eba1c39404071969e21ddd5abef69 (DEFAULT_TABLE table=test-workload [id=3cc1488e248b4e99a0705e301f54d797]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260810 22:05:31.024361 3067 tablet_service.cc:1467] Processing CreateTablet for tablet 159f82c64cc8402fb2570de1b5223e78 (DEFAULT_TABLE table=test-workload [id=3cc1488e248b4e99a0705e301f54d797]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260810 22:05:31.035030 3658 tablet_service.cc:1467] Processing CreateTablet for tablet 391eba1c39404071969e21ddd5abef69 (DEFAULT_TABLE table=test-workload [id=3cc1488e248b4e99a0705e301f54d797]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260810 22:05:31.035732 3657 tablet_service.cc:1467] Processing CreateTablet for tablet 943ec2cbc29e44e3bcbe0e1e5b039f55 (DEFAULT_TABLE table=test-workload [id=3cc1488e248b4e99a0705e301f54d797]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260810 22:05:31.039729 3358 tablet_service.cc:1467] Processing CreateTablet for tablet cfb52501fc724335a0c3d3e6a7ee95db (DEFAULT_TABLE table=test-workload [id=3cc1488e248b4e99a0705e301f54d797]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260810 22:05:31.040830 3354 tablet_service.cc:1467] Processing CreateTablet for tablet 159f82c64cc8402fb2570de1b5223e78 (DEFAULT_TABLE table=test-workload [id=3cc1488e248b4e99a0705e301f54d797]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260810 22:05:31.040644 3357 tablet_service.cc:1467] Processing CreateTablet for tablet 943ec2cbc29e44e3bcbe0e1e5b039f55 (DEFAULT_TABLE table=test-workload [id=3cc1488e248b4e99a0705e301f54d797]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260810 22:05:31.039767 3355 tablet_service.cc:1467] Processing CreateTablet for tablet 7230e7ad4d70476e85b798e58579d61f (DEFAULT_TABLE table=test-workload [id=3cc1488e248b4e99a0705e301f54d797]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260810 22:05:31.125286 3782 tablet_bootstrap.cc:492] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a: Bootstrap starting.
I20260810 22:05:31.130890 3783 tablet_bootstrap.cc:492] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a: Bootstrap starting.
I20260810 22:05:31.140496 3784 tablet_bootstrap.cc:492] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a: Bootstrap starting.
I20260810 22:05:31.147357 3789 tablet_bootstrap.cc:492] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9: Bootstrap starting.
I20260810 22:05:31.147440 3781 tablet_bootstrap.cc:492] T 7230e7ad4d70476e85b798e58579d61f P bdb9b70cb73a412ab0d4049695126b0a: Bootstrap starting.
I20260810 22:05:31.152268 3790 tablet_bootstrap.cc:492] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9: Bootstrap starting.
I20260810 22:05:31.154168 3787 tablet_bootstrap.cc:492] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1: Bootstrap starting.
I20260810 22:05:31.157938 3785 tablet_bootstrap.cc:492] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1: Bootstrap starting.
I20260810 22:05:31.161021 3784 tablet_bootstrap.cc:654] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:31.161090 3792 tablet_bootstrap.cc:492] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1: Bootstrap starting.
I20260810 22:05:31.164322 3793 tablet_bootstrap.cc:492] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9: Bootstrap starting.
I20260810 22:05:31.168955 3789 tablet_bootstrap.cc:654] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:31.170468 3793 tablet_bootstrap.cc:654] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:31.173264 3793 log.cc:824] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9: Log is configured to *not* fsync() on all Append() calls
I20260810 22:05:31.174196 3784 log.cc:824] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a: Log is configured to *not* fsync() on all Append() calls
I20260810 22:05:31.176288 3782 tablet_bootstrap.cc:654] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:31.178819 3790 tablet_bootstrap.cc:654] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:31.184404 3792 tablet_bootstrap.cc:654] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:31.186726 3787 tablet_bootstrap.cc:654] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:31.190418 3794 tablet_bootstrap.cc:492] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9: Bootstrap starting.
I20260810 22:05:31.195096 3785 tablet_bootstrap.cc:654] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:31.196760 3783 tablet_bootstrap.cc:654] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:31.215920 3798 tablet_bootstrap.cc:492] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490: Bootstrap starting.
I20260810 22:05:31.202198 3792 log.cc:824] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1: Log is configured to *not* fsync() on all Append() calls
I20260810 22:05:31.227197 3784 tablet_bootstrap.cc:492] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a: No bootstrap required, opened a new log
I20260810 22:05:31.227979 3784 ts_tablet_manager.cc:1397] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a: Time spent bootstrapping tablet: real 0.088s user 0.013s sys 0.014s
I20260810 22:05:31.254396 3802 tablet_bootstrap.cc:492] T 7230e7ad4d70476e85b798e58579d61f P 53b64278bdc549e9a56784b6b3295490: Bootstrap starting.
I20260810 22:05:31.257373 3803 tablet_bootstrap.cc:492] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490: Bootstrap starting.
I20260810 22:05:31.260448 3793 tablet_bootstrap.cc:492] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9: No bootstrap required, opened a new log
I20260810 22:05:31.261230 3793 ts_tablet_manager.cc:1397] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9: Time spent bootstrapping tablet: real 0.097s user 0.024s sys 0.018s
I20260810 22:05:31.293921 3785 tablet_bootstrap.cc:492] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1: No bootstrap required, opened a new log
I20260810 22:05:31.295470 3785 ts_tablet_manager.cc:1397] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1: Time spent bootstrapping tablet: real 0.138s user 0.008s sys 0.021s
I20260810 22:05:31.301723 3792 tablet_bootstrap.cc:492] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1: No bootstrap required, opened a new log
I20260810 22:05:31.302439 3792 ts_tablet_manager.cc:1397] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1: Time spent bootstrapping tablet: real 0.143s user 0.011s sys 0.019s
I20260810 22:05:31.303038 3811 tablet_bootstrap.cc:492] T 159f82c64cc8402fb2570de1b5223e78 P 53b64278bdc549e9a56784b6b3295490: Bootstrap starting.
I20260810 22:05:31.359213 3787 tablet_bootstrap.cc:492] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1: No bootstrap required, opened a new log
I20260810 22:05:31.359949 3787 ts_tablet_manager.cc:1397] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1: Time spent bootstrapping tablet: real 0.206s user 0.018s sys 0.017s
I20260810 22:05:31.362185 3790 tablet_bootstrap.cc:492] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9: No bootstrap required, opened a new log
I20260810 22:05:31.362912 3790 ts_tablet_manager.cc:1397] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9: Time spent bootstrapping tablet: real 0.212s user 0.026s sys 0.019s
I20260810 22:05:31.368019 3781 tablet_bootstrap.cc:654] T 7230e7ad4d70476e85b798e58579d61f P bdb9b70cb73a412ab0d4049695126b0a: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:31.381110 3794 tablet_bootstrap.cc:654] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:31.383711 3789 tablet_bootstrap.cc:492] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9: No bootstrap required, opened a new log
I20260810 22:05:31.384469 3789 ts_tablet_manager.cc:1397] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9: Time spent bootstrapping tablet: real 0.238s user 0.024s sys 0.090s
I20260810 22:05:31.409629 3785 raft_consensus.cc:348] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:31.410893 3785 raft_consensus.cc:374] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:31.411305 3785 raft_consensus.cc:729] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da5bd62264364fd198587af606253dd1, State: Initialized, Role: FOLLOWER
I20260810 22:05:31.412354 3787 raft_consensus.cc:348] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:31.413575 3787 raft_consensus.cc:374] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:31.414137 3787 raft_consensus.cc:729] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da5bd62264364fd198587af606253dd1, State: Initialized, Role: FOLLOWER
I20260810 22:05:31.416239 3787 consensus_queue.cc:260] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:31.423754 3802 tablet_bootstrap.cc:654] T 7230e7ad4d70476e85b798e58579d61f P 53b64278bdc549e9a56784b6b3295490: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:31.426117 3792 raft_consensus.cc:348] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } }
I20260810 22:05:31.427314 3792 raft_consensus.cc:374] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:31.427816 3792 raft_consensus.cc:729] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da5bd62264364fd198587af606253dd1, State: Initialized, Role: FOLLOWER
I20260810 22:05:31.435921 3787 ts_tablet_manager.cc:1428] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1: Time spent starting tablet: real 0.076s user 0.008s sys 0.007s
I20260810 22:05:31.419703 3785 consensus_queue.cc:260] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:31.437536 3794 tablet_bootstrap.cc:492] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9: No bootstrap required, opened a new log
I20260810 22:05:31.438519 3794 ts_tablet_manager.cc:1397] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9: Time spent bootstrapping tablet: real 0.252s user 0.016s sys 0.023s
I20260810 22:05:31.442353 3748 heartbeater.cc:503] Master 127.27.17.190:33447 was elected leader, sending a full tablet report...
I20260810 22:05:31.441501 3793 raft_consensus.cc:348] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } }
I20260810 22:05:31.443188 3793 raft_consensus.cc:374] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:31.443874 3793 raft_consensus.cc:729] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742853e1f3ae4ac1b86265b6adb193f9, State: Initialized, Role: FOLLOWER
I20260810 22:05:31.445989 3782 tablet_bootstrap.cc:492] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a: No bootstrap required, opened a new log
I20260810 22:05:31.446836 3783 tablet_bootstrap.cc:492] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a: No bootstrap required, opened a new log
I20260810 22:05:31.447515 3783 ts_tablet_manager.cc:1397] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a: Time spent bootstrapping tablet: real 0.317s user 0.008s sys 0.020s
I20260810 22:05:31.447760 3782 ts_tablet_manager.cc:1397] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a: Time spent bootstrapping tablet: real 0.323s user 0.008s sys 0.021s
I20260810 22:05:31.429209 3792 consensus_queue.cc:260] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } }
I20260810 22:05:31.445739 3790 raft_consensus.cc:348] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:31.455163 3790 raft_consensus.cc:374] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:31.455773 3790 raft_consensus.cc:729] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742853e1f3ae4ac1b86265b6adb193f9, State: Initialized, Role: FOLLOWER
I20260810 22:05:31.431629 3803 tablet_bootstrap.cc:654] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:31.456018 3798 tablet_bootstrap.cc:654] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:31.457890 3793 consensus_queue.cc:260] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } }
I20260810 22:05:31.457244 3790 consensus_queue.cc:260] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:31.464506 3792 ts_tablet_manager.cc:1428] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1: Time spent starting tablet: real 0.161s user 0.017s sys 0.006s
I20260810 22:05:31.463240 3789 raft_consensus.cc:348] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } }
I20260810 22:05:31.469053 3789 raft_consensus.cc:374] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:31.469594 3789 raft_consensus.cc:729] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742853e1f3ae4ac1b86265b6adb193f9, State: Initialized, Role: FOLLOWER
I20260810 22:05:31.470852 3789 consensus_queue.cc:260] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } }
I20260810 22:05:31.461268 3794 raft_consensus.cc:348] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:31.474694 3794 raft_consensus.cc:374] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:31.477077 3781 tablet_bootstrap.cc:492] T 7230e7ad4d70476e85b798e58579d61f P bdb9b70cb73a412ab0d4049695126b0a: No bootstrap required, opened a new log
I20260810 22:05:31.477607 3781 ts_tablet_manager.cc:1397] T 7230e7ad4d70476e85b798e58579d61f P bdb9b70cb73a412ab0d4049695126b0a: Time spent bootstrapping tablet: real 0.331s user 0.022s sys 0.017s
I20260810 22:05:31.475015 3794 raft_consensus.cc:729] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742853e1f3ae4ac1b86265b6adb193f9, State: Initialized, Role: FOLLOWER
I20260810 22:05:31.481668 3785 ts_tablet_manager.cc:1428] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1: Time spent starting tablet: real 0.185s user 0.033s sys 0.032s
I20260810 22:05:31.482016 3794 consensus_queue.cc:260] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:31.492093 3783 raft_consensus.cc:348] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } }
I20260810 22:05:31.493505 3783 raft_consensus.cc:374] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:31.494046 3783 raft_consensus.cc:729] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdb9b70cb73a412ab0d4049695126b0a, State: Initialized, Role: FOLLOWER
I20260810 22:05:31.495724 3793 ts_tablet_manager.cc:1428] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9: Time spent starting tablet: real 0.225s user 0.040s sys 0.030s
I20260810 22:05:31.496021 3782 raft_consensus.cc:348] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } }
I20260810 22:05:31.497298 3782 raft_consensus.cc:374] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:31.497851 3782 raft_consensus.cc:729] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdb9b70cb73a412ab0d4049695126b0a, State: Initialized, Role: FOLLOWER
I20260810 22:05:31.499559 3782 consensus_queue.cc:260] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } }
I20260810 22:05:31.501919 3783 consensus_queue.cc:260] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } }
I20260810 22:05:31.508033 3790 ts_tablet_manager.cc:1428] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9: Time spent starting tablet: real 0.145s user 0.013s sys 0.008s
I20260810 22:05:31.513821 3824 raft_consensus.cc:482] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:31.516882 3794 ts_tablet_manager.cc:1428] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9: Time spent starting tablet: real 0.077s user 0.008s sys 0.009s
I20260810 22:05:31.514884 3824 raft_consensus.cc:504] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:31.521850 3789 ts_tablet_manager.cc:1428] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9: Time spent starting tablet: real 0.137s user 0.009s sys 0.031s
I20260810 22:05:31.527897 3824 leader_election.cc:288] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53b64278bdc549e9a56784b6b3295490 (127.27.17.130:44557), da5bd62264364fd198587af606253dd1 (127.27.17.132:41969)
I20260810 22:05:31.515462 3781 raft_consensus.cc:348] T 7230e7ad4d70476e85b798e58579d61f P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } }
I20260810 22:05:31.534698 3781 raft_consensus.cc:374] T 7230e7ad4d70476e85b798e58579d61f P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:31.535266 3781 raft_consensus.cc:729] T 7230e7ad4d70476e85b798e58579d61f P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdb9b70cb73a412ab0d4049695126b0a, State: Initialized, Role: FOLLOWER
I20260810 22:05:31.548421 3802 log.cc:824] T 7230e7ad4d70476e85b798e58579d61f P 53b64278bdc549e9a56784b6b3295490: Log is configured to *not* fsync() on all Append() calls
W20260810 22:05:31.550314 3749 tablet.cc:2369] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:05:31.551837 3784 raft_consensus.cc:348] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:31.554322 3781 consensus_queue.cc:260] T 7230e7ad4d70476e85b798e58579d61f P bdb9b70cb73a412ab0d4049695126b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } }
I20260810 22:05:31.558044 3823 raft_consensus.cc:482] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:31.559157 3823 raft_consensus.cc:504] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:31.574376 3823 leader_election.cc:288] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131:45861), 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615)
W20260810 22:05:31.610575 2860 tablet.cc:2369] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:05:31.553217 3784 raft_consensus.cc:374] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:31.583464 3811 tablet_bootstrap.cc:654] T 159f82c64cc8402fb2570de1b5223e78 P 53b64278bdc549e9a56784b6b3295490: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:31.618975 3784 raft_consensus.cc:729] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdb9b70cb73a412ab0d4049695126b0a, State: Initialized, Role: FOLLOWER
I20260810 22:05:31.630493 3784 consensus_queue.cc:260] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:31.641911 3784 ts_tablet_manager.cc:1428] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a: Time spent starting tablet: real 0.413s user 0.042s sys 0.037s
I20260810 22:05:31.642465 3782 ts_tablet_manager.cc:1428] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a: Time spent starting tablet: real 0.194s user 0.003s sys 0.019s
I20260810 22:05:31.650738 3829 raft_consensus.cc:482] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:31.657219 3823 raft_consensus.cc:482] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:31.658002 3823 raft_consensus.cc:504] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } }
I20260810 22:05:31.660624 3823 leader_election.cc:288] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53b64278bdc549e9a56784b6b3295490 (127.27.17.130:44557), bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131:45861)
I20260810 22:05:31.687973 3798 tablet_bootstrap.cc:492] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490: No bootstrap required, opened a new log
I20260810 22:05:31.688783 3798 ts_tablet_manager.cc:1397] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490: Time spent bootstrapping tablet: real 0.474s user 0.013s sys 0.027s
I20260810 22:05:31.651978 3829 raft_consensus.cc:504] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } }
I20260810 22:05:31.692474 3781 ts_tablet_manager.cc:1428] T 7230e7ad4d70476e85b798e58579d61f P bdb9b70cb73a412ab0d4049695126b0a: Time spent starting tablet: real 0.215s user 0.006s sys 0.023s
I20260810 22:05:31.713745 3783 ts_tablet_manager.cc:1428] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a: Time spent starting tablet: real 0.266s user 0.013s sys 0.014s
I20260810 22:05:31.716749 3829 leader_election.cc:288] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615), 53b64278bdc549e9a56784b6b3295490 (127.27.17.130:44557)
I20260810 22:05:31.726742 3803 tablet_bootstrap.cc:492] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490: No bootstrap required, opened a new log
I20260810 22:05:31.730651 3803 ts_tablet_manager.cc:1397] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490: Time spent bootstrapping tablet: real 0.474s user 0.010s sys 0.032s
I20260810 22:05:31.730167 3385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943ec2cbc29e44e3bcbe0e1e5b039f55" candidate_uuid: "da5bd62264364fd198587af606253dd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdb9b70cb73a412ab0d4049695126b0a" is_pre_election: true
I20260810 22:05:31.743674 3802 tablet_bootstrap.cc:492] T 7230e7ad4d70476e85b798e58579d61f P 53b64278bdc549e9a56784b6b3295490: No bootstrap required, opened a new log
I20260810 22:05:31.745615 3802 ts_tablet_manager.cc:1397] T 7230e7ad4d70476e85b798e58579d61f P 53b64278bdc549e9a56784b6b3295490: Time spent bootstrapping tablet: real 0.492s user 0.018s sys 0.023s
I20260810 22:05:31.733573 3384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7230e7ad4d70476e85b798e58579d61f" candidate_uuid: "da5bd62264364fd198587af606253dd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdb9b70cb73a412ab0d4049695126b0a" is_pre_election: true
I20260810 22:05:31.749719 3829 raft_consensus.cc:482] T 7230e7ad4d70476e85b798e58579d61f P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:31.755872 3829 raft_consensus.cc:504] T 7230e7ad4d70476e85b798e58579d61f P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } }
I20260810 22:05:31.760149 3829 leader_election.cc:288] T 7230e7ad4d70476e85b798e58579d61f P bdb9b70cb73a412ab0d4049695126b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53b64278bdc549e9a56784b6b3295490 (127.27.17.130:44557), da5bd62264364fd198587af606253dd1 (127.27.17.132:41969)
I20260810 22:05:31.766664 3384 raft_consensus.cc:2454] T 7230e7ad4d70476e85b798e58579d61f P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da5bd62264364fd198587af606253dd1 in term 0.
I20260810 22:05:31.768100 3385 raft_consensus.cc:2454] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da5bd62264364fd198587af606253dd1 in term 0.
I20260810 22:05:31.775231 3824 raft_consensus.cc:482] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:31.776854 3824 raft_consensus.cc:504] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:31.783419 3824 leader_election.cc:288] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131:45861), da5bd62264364fd198587af606253dd1 (127.27.17.132:41969)
I20260810 22:05:31.791083 3811 tablet_bootstrap.cc:492] T 159f82c64cc8402fb2570de1b5223e78 P 53b64278bdc549e9a56784b6b3295490: No bootstrap required, opened a new log
I20260810 22:05:31.791998 3811 ts_tablet_manager.cc:1397] T 159f82c64cc8402fb2570de1b5223e78 P 53b64278bdc549e9a56784b6b3295490: Time spent bootstrapping tablet: real 0.490s user 0.015s sys 0.035s
I20260810 22:05:31.812044 3599 leader_election.cc:302] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdb9b70cb73a412ab0d4049695126b0a, da5bd62264364fd198587af606253dd1; no voters:
I20260810 22:05:31.814437 3599 leader_election.cc:302] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdb9b70cb73a412ab0d4049695126b0a, da5bd62264364fd198587af606253dd1; no voters:
W20260810 22:05:31.817422 3451 tablet.cc:2369] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:05:31.817670 3822 raft_consensus.cc:2781] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:31.818409 3822 raft_consensus.cc:482] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:31.819031 3822 raft_consensus.cc:3037] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:31.804697 3099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "391eba1c39404071969e21ddd5abef69" candidate_uuid: "742853e1f3ae4ac1b86265b6adb193f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53b64278bdc549e9a56784b6b3295490" is_pre_election: true
I20260810 22:05:31.836864 3824 raft_consensus.cc:482] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:31.838146 3824 raft_consensus.cc:504] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } }
I20260810 22:05:31.821558 3823 raft_consensus.cc:2781] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:31.839915 3823 raft_consensus.cc:482] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:31.840490 3823 raft_consensus.cc:3037] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:31.842207 3824 leader_election.cc:288] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131:45861), 53b64278bdc549e9a56784b6b3295490 (127.27.17.130:44557)
W20260810 22:05:31.843917 2710 leader_election.cc:341] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53b64278bdc549e9a56784b6b3295490 (127.27.17.130:44557): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:05:31.848258 3690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "391eba1c39404071969e21ddd5abef69" candidate_uuid: "742853e1f3ae4ac1b86265b6adb193f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da5bd62264364fd198587af606253dd1" is_pre_election: true
I20260810 22:05:31.849152 3690 raft_consensus.cc:2454] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 742853e1f3ae4ac1b86265b6adb193f9 in term 0.
I20260810 22:05:31.850349 3689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943ec2cbc29e44e3bcbe0e1e5b039f55" candidate_uuid: "742853e1f3ae4ac1b86265b6adb193f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da5bd62264364fd198587af606253dd1" is_pre_election: true
I20260810 22:05:31.850481 3099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb52501fc724335a0c3d3e6a7ee95db" candidate_uuid: "742853e1f3ae4ac1b86265b6adb193f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53b64278bdc549e9a56784b6b3295490" is_pre_election: true
I20260810 22:05:31.857646 2708 leader_election.cc:302] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 742853e1f3ae4ac1b86265b6adb193f9, da5bd62264364fd198587af606253dd1; no voters: 53b64278bdc549e9a56784b6b3295490
I20260810 22:05:31.858876 3802 raft_consensus.cc:348] T 7230e7ad4d70476e85b798e58579d61f P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } }
I20260810 22:05:31.858793 3803 raft_consensus.cc:348] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } }
I20260810 22:05:31.859932 3802 raft_consensus.cc:374] T 7230e7ad4d70476e85b798e58579d61f P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:31.860013 3803 raft_consensus.cc:374] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:31.860412 3803 raft_consensus.cc:729] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b64278bdc549e9a56784b6b3295490, State: Initialized, Role: FOLLOWER
I20260810 22:05:31.861372 3099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7230e7ad4d70476e85b798e58579d61f" candidate_uuid: "da5bd62264364fd198587af606253dd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53b64278bdc549e9a56784b6b3295490" is_pre_election: true
I20260810 22:05:31.860280 3802 raft_consensus.cc:729] T 7230e7ad4d70476e85b798e58579d61f P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b64278bdc549e9a56784b6b3295490, State: Initialized, Role: FOLLOWER
I20260810 22:05:31.861858 3384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943ec2cbc29e44e3bcbe0e1e5b039f55" candidate_uuid: "742853e1f3ae4ac1b86265b6adb193f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdb9b70cb73a412ab0d4049695126b0a" is_pre_election: true
I20260810 22:05:31.862639 3384 raft_consensus.cc:2454] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 742853e1f3ae4ac1b86265b6adb193f9 in term 0.
I20260810 22:05:31.862810 3802 consensus_queue.cc:260] T 7230e7ad4d70476e85b798e58579d61f P 53b64278bdc549e9a56784b6b3295490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } }
I20260810 22:05:31.864164 3099 raft_consensus.cc:2454] T 7230e7ad4d70476e85b798e58579d61f P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da5bd62264364fd198587af606253dd1 in term 0.
I20260810 22:05:31.868422 3803 consensus_queue.cc:260] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } }
I20260810 22:05:31.871852 3811 raft_consensus.cc:348] T 159f82c64cc8402fb2570de1b5223e78 P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } }
I20260810 22:05:31.872944 3811 raft_consensus.cc:374] T 159f82c64cc8402fb2570de1b5223e78 P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:31.873361 3811 raft_consensus.cc:729] T 159f82c64cc8402fb2570de1b5223e78 P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b64278bdc549e9a56784b6b3295490, State: Initialized, Role: FOLLOWER
I20260810 22:05:31.874284 3811 consensus_queue.cc:260] T 159f82c64cc8402fb2570de1b5223e78 P 53b64278bdc549e9a56784b6b3295490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } }
I20260810 22:05:31.875660 3824 raft_consensus.cc:2781] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:31.876223 3824 raft_consensus.cc:482] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:31.876314 3822 raft_consensus.cc:504] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:31.876709 3824 raft_consensus.cc:3037] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:31.881911 3798 raft_consensus.cc:348] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:31.884290 3798 raft_consensus.cc:374] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:31.884776 3798 raft_consensus.cc:729] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b64278bdc549e9a56784b6b3295490, State: Initialized, Role: FOLLOWER
I20260810 22:05:31.885625 3798 consensus_queue.cc:260] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:31.877300 3802 ts_tablet_manager.cc:1428] T 7230e7ad4d70476e85b798e58579d61f P 53b64278bdc549e9a56784b6b3295490: Time spent starting tablet: real 0.125s user 0.024s sys 0.017s
W20260810 22:05:31.893913 2710 leader_election.cc:341] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53b64278bdc549e9a56784b6b3295490 (127.27.17.130:44557): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:05:31.894400 3384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb52501fc724335a0c3d3e6a7ee95db" candidate_uuid: "742853e1f3ae4ac1b86265b6adb193f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdb9b70cb73a412ab0d4049695126b0a" is_pre_election: true
I20260810 22:05:31.894636 2710 leader_election.cc:302] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 742853e1f3ae4ac1b86265b6adb193f9, bdb9b70cb73a412ab0d4049695126b0a; no voters:
I20260810 22:05:31.895026 3384 raft_consensus.cc:2454] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 742853e1f3ae4ac1b86265b6adb193f9 in term 0.
I20260810 22:05:31.896031 3825 raft_consensus.cc:2781] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:31.896385 3825 raft_consensus.cc:482] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:31.896744 3825 raft_consensus.cc:3037] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:31.900513 3863 raft_consensus.cc:482] T 159f82c64cc8402fb2570de1b5223e78 P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:31.900595 3824 raft_consensus.cc:504] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:31.900970 3863 raft_consensus.cc:504] T 159f82c64cc8402fb2570de1b5223e78 P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } }
I20260810 22:05:31.904147 3825 raft_consensus.cc:504] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:31.906334 3825 leader_election.cc:288] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 [CANDIDATE]: Term 1 election: Requested vote from peers bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131:45861), da5bd62264364fd198587af606253dd1 (127.27.17.132:41969)
I20260810 22:05:31.906541 3689 raft_consensus.cc:2379] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 742853e1f3ae4ac1b86265b6adb193f9 in current term 1: Already voted for candidate da5bd62264364fd198587af606253dd1 in this term.
I20260810 22:05:31.907722 3863 leader_election.cc:288] T 159f82c64cc8402fb2570de1b5223e78 P 53b64278bdc549e9a56784b6b3295490 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615), bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131:45861)
I20260810 22:05:31.910087 3803 ts_tablet_manager.cc:1428] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490: Time spent starting tablet: real 0.179s user 0.011s sys 0.003s
I20260810 22:05:31.910369 3384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943ec2cbc29e44e3bcbe0e1e5b039f55" candidate_uuid: "742853e1f3ae4ac1b86265b6adb193f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdb9b70cb73a412ab0d4049695126b0a"
I20260810 22:05:31.910928 3384 raft_consensus.cc:3037] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:31.910971 3688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943ec2cbc29e44e3bcbe0e1e5b039f55" candidate_uuid: "742853e1f3ae4ac1b86265b6adb193f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da5bd62264364fd198587af606253dd1"
I20260810 22:05:31.913998 3798 ts_tablet_manager.cc:1428] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490: Time spent starting tablet: real 0.225s user 0.020s sys 0.023s
I20260810 22:05:31.882920 3823 raft_consensus.cc:504] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } }
I20260810 22:05:31.897598 3690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7230e7ad4d70476e85b798e58579d61f" candidate_uuid: "bdb9b70cb73a412ab0d4049695126b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da5bd62264364fd198587af606253dd1" is_pre_election: true
I20260810 22:05:31.918565 3690 raft_consensus.cc:2379] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bdb9b70cb73a412ab0d4049695126b0a in current term 1: Already voted for candidate da5bd62264364fd198587af606253dd1 in this term.
I20260810 22:05:31.918738 3384 raft_consensus.cc:2454] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 742853e1f3ae4ac1b86265b6adb193f9 in term 1.
I20260810 22:05:31.919760 3825 raft_consensus.cc:482] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:31.921020 3825 raft_consensus.cc:504] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } }
W20260810 22:05:31.923486 3160 tablet.cc:2369] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:05:31.926267 3384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7230e7ad4d70476e85b798e58579d61f" candidate_uuid: "da5bd62264364fd198587af606253dd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdb9b70cb73a412ab0d4049695126b0a"
I20260810 22:05:31.923606 3811 ts_tablet_manager.cc:1428] T 159f82c64cc8402fb2570de1b5223e78 P 53b64278bdc549e9a56784b6b3295490: Time spent starting tablet: real 0.130s user 0.008s sys 0.011s
I20260810 22:05:31.928352 3825 leader_election.cc:288] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53b64278bdc549e9a56784b6b3295490 (127.27.17.130:44557), bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131:45861)
I20260810 22:05:31.928736 3384 raft_consensus.cc:3037] T 7230e7ad4d70476e85b798e58579d61f P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:31.930740 3690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "391eba1c39404071969e21ddd5abef69" candidate_uuid: "742853e1f3ae4ac1b86265b6adb193f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da5bd62264364fd198587af606253dd1"
I20260810 22:05:31.931515 3690 raft_consensus.cc:3037] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:31.935820 3384 raft_consensus.cc:2454] T 7230e7ad4d70476e85b798e58579d61f P bdb9b70cb73a412ab0d4049695126b0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da5bd62264364fd198587af606253dd1 in term 1.
I20260810 22:05:31.936563 3099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7230e7ad4d70476e85b798e58579d61f" candidate_uuid: "da5bd62264364fd198587af606253dd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53b64278bdc549e9a56784b6b3295490"
I20260810 22:05:31.929977 2710 leader_election.cc:302] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 742853e1f3ae4ac1b86265b6adb193f9, bdb9b70cb73a412ab0d4049695126b0a; no voters: 53b64278bdc549e9a56784b6b3295490
I20260810 22:05:31.937168 3099 raft_consensus.cc:3037] T 7230e7ad4d70476e85b798e58579d61f P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:31.937038 3599 leader_election.cc:302] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdb9b70cb73a412ab0d4049695126b0a, da5bd62264364fd198587af606253dd1; no voters:
I20260810 22:05:31.938215 2710 leader_election.cc:302] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 742853e1f3ae4ac1b86265b6adb193f9, bdb9b70cb73a412ab0d4049695126b0a; no voters: da5bd62264364fd198587af606253dd1
I20260810 22:05:31.941217 3823 leader_election.cc:288] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [CANDIDATE]: Term 1 election: Requested vote from peers 53b64278bdc549e9a56784b6b3295490 (127.27.17.130:44557), bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131:45861)
I20260810 22:05:31.941753 3823 raft_consensus.cc:2781] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:31.941771 3829 raft_consensus.cc:482] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:31.942200 3823 raft_consensus.cc:686] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [term 1 LEADER]: Becoming Leader. State: Replica: da5bd62264364fd198587af606253dd1, State: Running, Role: LEADER
I20260810 22:05:31.942305 3829 raft_consensus.cc:504] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } }
I20260810 22:05:31.943079 3823 consensus_queue.cc:237] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } }
I20260810 22:05:31.944442 3099 raft_consensus.cc:2454] T 7230e7ad4d70476e85b798e58579d61f P 53b64278bdc549e9a56784b6b3295490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da5bd62264364fd198587af606253dd1 in term 1.
I20260810 22:05:31.944665 3829 leader_election.cc:288] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615), 53b64278bdc549e9a56784b6b3295490 (127.27.17.130:44557)
I20260810 22:05:31.946552 3822 leader_election.cc:288] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [CANDIDATE]: Term 1 election: Requested vote from peers bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131:45861), 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615)
I20260810 22:05:31.950712 3690 raft_consensus.cc:2454] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 742853e1f3ae4ac1b86265b6adb193f9 in term 1.
I20260810 22:05:31.952412 3098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7230e7ad4d70476e85b798e58579d61f" candidate_uuid: "bdb9b70cb73a412ab0d4049695126b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53b64278bdc549e9a56784b6b3295490" is_pre_election: true
I20260810 22:05:31.953775 3098 raft_consensus.cc:2379] T 7230e7ad4d70476e85b798e58579d61f P 53b64278bdc549e9a56784b6b3295490 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bdb9b70cb73a412ab0d4049695126b0a in current term 1: Already voted for candidate da5bd62264364fd198587af606253dd1 in this term.
I20260810 22:05:31.955190 3098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159f82c64cc8402fb2570de1b5223e78" candidate_uuid: "bdb9b70cb73a412ab0d4049695126b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53b64278bdc549e9a56784b6b3295490" is_pre_election: true
I20260810 22:05:31.955849 3098 raft_consensus.cc:2454] T 159f82c64cc8402fb2570de1b5223e78 P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdb9b70cb73a412ab0d4049695126b0a in term 0.
I20260810 22:05:31.956887 3384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943ec2cbc29e44e3bcbe0e1e5b039f55" candidate_uuid: "da5bd62264364fd198587af606253dd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdb9b70cb73a412ab0d4049695126b0a"
I20260810 22:05:31.957029 3294 leader_election.cc:302] T 7230e7ad4d70476e85b798e58579d61f P bdb9b70cb73a412ab0d4049695126b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bdb9b70cb73a412ab0d4049695126b0a; no voters: 53b64278bdc549e9a56784b6b3295490, da5bd62264364fd198587af606253dd1
I20260810 22:05:31.951484 3099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb52501fc724335a0c3d3e6a7ee95db" candidate_uuid: "bdb9b70cb73a412ab0d4049695126b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53b64278bdc549e9a56784b6b3295490" is_pre_election: true
I20260810 22:05:31.957947 3829 raft_consensus.cc:2726] T 7230e7ad4d70476e85b798e58579d61f P bdb9b70cb73a412ab0d4049695126b0a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 22:05:31.958209 3099 raft_consensus.cc:2454] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdb9b70cb73a412ab0d4049695126b0a in term 0.
I20260810 22:05:31.958595 3384 raft_consensus.cc:2379] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate da5bd62264364fd198587af606253dd1 in current term 1: Already voted for candidate 742853e1f3ae4ac1b86265b6adb193f9 in this term.
I20260810 22:05:31.959175 3294 leader_election.cc:302] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53b64278bdc549e9a56784b6b3295490, bdb9b70cb73a412ab0d4049695126b0a; no voters:
I20260810 22:05:31.959844 3829 raft_consensus.cc:2781] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:31.960021 3294 leader_election.cc:302] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53b64278bdc549e9a56784b6b3295490, bdb9b70cb73a412ab0d4049695126b0a; no voters:
I20260810 22:05:31.960170 3829 raft_consensus.cc:482] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:31.960505 3829 raft_consensus.cc:3037] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:31.960843 3851 raft_consensus.cc:2781] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:31.961222 3851 raft_consensus.cc:482] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:31.961571 3851 raft_consensus.cc:3037] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:31.939944 3827 raft_consensus.cc:2781] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:31.968235 3851 raft_consensus.cc:504] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } }
I20260810 22:05:31.970117 3851 leader_election.cc:288] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [CANDIDATE]: Term 1 election: Requested vote from peers 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615), 53b64278bdc549e9a56784b6b3295490 (127.27.17.130:44557)
I20260810 22:05:31.971231 3099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb52501fc724335a0c3d3e6a7ee95db" candidate_uuid: "bdb9b70cb73a412ab0d4049695126b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53b64278bdc549e9a56784b6b3295490"
I20260810 22:05:31.971843 3099 raft_consensus.cc:3037] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:31.978140 3099 raft_consensus.cc:2454] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdb9b70cb73a412ab0d4049695126b0a in term 1.
I20260810 22:05:31.979246 3294 leader_election.cc:302] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53b64278bdc549e9a56784b6b3295490, bdb9b70cb73a412ab0d4049695126b0a; no voters:
I20260810 22:05:31.980012 3851 raft_consensus.cc:2781] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:31.979357 3827 raft_consensus.cc:482] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:31.980489 3827 raft_consensus.cc:3037] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:31.980737 3851 raft_consensus.cc:686] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [term 1 LEADER]: Becoming Leader. State: Replica: bdb9b70cb73a412ab0d4049695126b0a, State: Running, Role: LEADER
I20260810 22:05:31.981652 3851 consensus_queue.cc:237] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } }
I20260810 22:05:31.982003 2708 leader_election.cc:302] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 742853e1f3ae4ac1b86265b6adb193f9, da5bd62264364fd198587af606253dd1; no voters:
I20260810 22:05:31.983053 3829 raft_consensus.cc:504] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } }
I20260810 22:05:31.987300 3099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159f82c64cc8402fb2570de1b5223e78" candidate_uuid: "bdb9b70cb73a412ab0d4049695126b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53b64278bdc549e9a56784b6b3295490"
I20260810 22:05:31.987999 3099 raft_consensus.cc:3037] T 159f82c64cc8402fb2570de1b5223e78 P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:31.994113 3099 raft_consensus.cc:2454] T 159f82c64cc8402fb2570de1b5223e78 P 53b64278bdc549e9a56784b6b3295490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdb9b70cb73a412ab0d4049695126b0a in term 1.
I20260810 22:05:31.997671 3829 leader_election.cc:288] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a [CANDIDATE]: Term 1 election: Requested vote from peers 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615), 53b64278bdc549e9a56784b6b3295490 (127.27.17.130:44557)
I20260810 22:05:31.998806 3294 leader_election.cc:302] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53b64278bdc549e9a56784b6b3295490, bdb9b70cb73a412ab0d4049695126b0a; no voters:
I20260810 22:05:32.004061 2801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb52501fc724335a0c3d3e6a7ee95db" candidate_uuid: "bdb9b70cb73a412ab0d4049695126b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "742853e1f3ae4ac1b86265b6adb193f9" is_pre_election: true
I20260810 22:05:31.929265 3824 leader_election.cc:288] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [CANDIDATE]: Term 1 election: Requested vote from peers 53b64278bdc549e9a56784b6b3295490 (127.27.17.130:44557), da5bd62264364fd198587af606253dd1 (127.27.17.132:41969)
I20260810 22:05:32.005659 3826 raft_consensus.cc:2781] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:32.006474 3826 raft_consensus.cc:686] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 LEADER]: Becoming Leader. State: Replica: 742853e1f3ae4ac1b86265b6adb193f9, State: Running, Role: LEADER
I20260810 22:05:32.005949 2474 catalog_manager.cc:5495] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 reported cstate change: term changed from 0 to 1, leader changed from <none> to da5bd62264364fd198587af606253dd1 (127.27.17.132). New cstate: current_term: 1 leader_uuid: "da5bd62264364fd198587af606253dd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } health_report { overall_health: HEALTHY } } }
I20260810 22:05:32.007459 3826 consensus_queue.cc:237] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:32.009114 3829 raft_consensus.cc:2781] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:32.010109 3829 raft_consensus.cc:686] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a [term 1 LEADER]: Becoming Leader. State: Replica: bdb9b70cb73a412ab0d4049695126b0a, State: Running, Role: LEADER
I20260810 22:05:32.010963 3829 consensus_queue.cc:237] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } }
I20260810 22:05:32.014460 3384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159f82c64cc8402fb2570de1b5223e78" candidate_uuid: "53b64278bdc549e9a56784b6b3295490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdb9b70cb73a412ab0d4049695126b0a" is_pre_election: true
I20260810 22:05:32.014950 2475 catalog_manager.cc:5495] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a reported cstate change: term changed from 0 to 1, leader changed from <none> to bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131). New cstate: current_term: 1 leader_uuid: "bdb9b70cb73a412ab0d4049695126b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } health_report { overall_health: UNKNOWN } } }
I20260810 22:05:32.005270 3824 raft_consensus.cc:2781] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 FOLLOWER]: Leader election won for term 1
W20260810 22:05:32.004451 2710 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.17.131:45861, user_credentials={real_user=slave}} blocked reactor thread for 66316.6us
I20260810 22:05:32.019966 3824 raft_consensus.cc:686] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 LEADER]: Becoming Leader. State: Replica: 742853e1f3ae4ac1b86265b6adb193f9, State: Running, Role: LEADER
I20260810 22:05:32.030548 3099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159f82c64cc8402fb2570de1b5223e78" candidate_uuid: "742853e1f3ae4ac1b86265b6adb193f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53b64278bdc549e9a56784b6b3295490" is_pre_election: true
I20260810 22:05:32.031388 3099 raft_consensus.cc:2379] T 159f82c64cc8402fb2570de1b5223e78 P 53b64278bdc549e9a56784b6b3295490 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 742853e1f3ae4ac1b86265b6adb193f9 in current term 1: Already voted for candidate bdb9b70cb73a412ab0d4049695126b0a in this term.
I20260810 22:05:32.033804 3384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159f82c64cc8402fb2570de1b5223e78" candidate_uuid: "742853e1f3ae4ac1b86265b6adb193f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdb9b70cb73a412ab0d4049695126b0a" is_pre_election: true
I20260810 22:05:32.037555 2800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159f82c64cc8402fb2570de1b5223e78" candidate_uuid: "bdb9b70cb73a412ab0d4049695126b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "742853e1f3ae4ac1b86265b6adb193f9" is_pre_election: true
I20260810 22:05:32.038587 2800 raft_consensus.cc:2454] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdb9b70cb73a412ab0d4049695126b0a in term 0.
I20260810 22:05:32.039940 2798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159f82c64cc8402fb2570de1b5223e78" candidate_uuid: "bdb9b70cb73a412ab0d4049695126b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "742853e1f3ae4ac1b86265b6adb193f9"
I20260810 22:05:32.040647 2798 raft_consensus.cc:3037] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:32.041436 2799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb52501fc724335a0c3d3e6a7ee95db" candidate_uuid: "bdb9b70cb73a412ab0d4049695126b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "742853e1f3ae4ac1b86265b6adb193f9"
I20260810 22:05:32.044872 3824 consensus_queue.cc:237] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:32.048642 3099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "391eba1c39404071969e21ddd5abef69" candidate_uuid: "742853e1f3ae4ac1b86265b6adb193f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53b64278bdc549e9a56784b6b3295490"
I20260810 22:05:32.049247 3099 raft_consensus.cc:3037] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:32.053768 2800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943ec2cbc29e44e3bcbe0e1e5b039f55" candidate_uuid: "da5bd62264364fd198587af606253dd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "742853e1f3ae4ac1b86265b6adb193f9"
I20260810 22:05:32.054991 2799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943ec2cbc29e44e3bcbe0e1e5b039f55" candidate_uuid: "da5bd62264364fd198587af606253dd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "742853e1f3ae4ac1b86265b6adb193f9" is_pre_election: true
I20260810 22:05:32.055414 3099 raft_consensus.cc:2454] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 742853e1f3ae4ac1b86265b6adb193f9 in term 1.
I20260810 22:05:32.056387 2710 leader_election.cc:302] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 742853e1f3ae4ac1b86265b6adb193f9; no voters: 53b64278bdc549e9a56784b6b3295490, bdb9b70cb73a412ab0d4049695126b0a
I20260810 22:05:32.075551 3599 leader_election.cc:302] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da5bd62264364fd198587af606253dd1; no voters: 742853e1f3ae4ac1b86265b6adb193f9, bdb9b70cb73a412ab0d4049695126b0a
I20260810 22:05:32.076647 3823 raft_consensus.cc:2726] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 22:05:32.082126 2798 raft_consensus.cc:2454] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdb9b70cb73a412ab0d4049695126b0a in term 1.
I20260810 22:05:32.082777 3825 raft_consensus.cc:2726] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 22:05:32.088454 3827 raft_consensus.cc:504] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } }
I20260810 22:05:32.090243 2801 raft_consensus.cc:2379] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bdb9b70cb73a412ab0d4049695126b0a in current term 1: Already voted for candidate 742853e1f3ae4ac1b86265b6adb193f9 in this term.
I20260810 22:05:32.096750 3384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb52501fc724335a0c3d3e6a7ee95db" candidate_uuid: "742853e1f3ae4ac1b86265b6adb193f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdb9b70cb73a412ab0d4049695126b0a"
I20260810 22:05:32.103636 3099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb52501fc724335a0c3d3e6a7ee95db" candidate_uuid: "742853e1f3ae4ac1b86265b6adb193f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53b64278bdc549e9a56784b6b3295490"
I20260810 22:05:32.104552 3099 raft_consensus.cc:2379] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 742853e1f3ae4ac1b86265b6adb193f9 in current term 1: Already voted for candidate bdb9b70cb73a412ab0d4049695126b0a in this term.
I20260810 22:05:32.107036 3827 leader_election.cc:288] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9 [CANDIDATE]: Term 1 election: Requested vote from peers bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131:45861), 53b64278bdc549e9a56784b6b3295490 (127.27.17.130:44557)
I20260810 22:05:32.107326 2710 leader_election.cc:302] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 742853e1f3ae4ac1b86265b6adb193f9; no voters: 53b64278bdc549e9a56784b6b3295490, bdb9b70cb73a412ab0d4049695126b0a
I20260810 22:05:32.102254 2801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159f82c64cc8402fb2570de1b5223e78" candidate_uuid: "53b64278bdc549e9a56784b6b3295490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "742853e1f3ae4ac1b86265b6adb193f9" is_pre_election: true
I20260810 22:05:32.108439 2801 raft_consensus.cc:2379] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 53b64278bdc549e9a56784b6b3295490 in current term 1: Already voted for candidate bdb9b70cb73a412ab0d4049695126b0a in this term.
I20260810 22:05:32.110239 3013 leader_election.cc:302] T 159f82c64cc8402fb2570de1b5223e78 P 53b64278bdc549e9a56784b6b3295490 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53b64278bdc549e9a56784b6b3295490; no voters: 742853e1f3ae4ac1b86265b6adb193f9, bdb9b70cb73a412ab0d4049695126b0a
I20260810 22:05:32.112095 3868 raft_consensus.cc:2726] T 159f82c64cc8402fb2570de1b5223e78 P 53b64278bdc549e9a56784b6b3295490 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 22:05:32.115350 2473 catalog_manager.cc:5495] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129). New cstate: current_term: 1 leader_uuid: "742853e1f3ae4ac1b86265b6adb193f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } health_report { overall_health: HEALTHY } } }
I20260810 22:05:32.120105 3827 raft_consensus.cc:2726] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 22:05:32.154853 2475 catalog_manager.cc:5495] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a reported cstate change: term changed from 0 to 1, leader changed from <none> to bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131). New cstate: current_term: 1 leader_uuid: "bdb9b70cb73a412ab0d4049695126b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } health_report { overall_health: HEALTHY } } }
I20260810 22:05:32.163324 2473 catalog_manager.cc:5495] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129). New cstate: current_term: 1 leader_uuid: "742853e1f3ae4ac1b86265b6adb193f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } health_report { overall_health: HEALTHY } } }
I20260810 22:05:32.360622 3823 consensus_queue.cc:1035] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:32.374488 3873 consensus_queue.cc:1035] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:32.469728 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 2563
I20260810 22:05:32.514464 3829 consensus_queue.cc:1035] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:32.535933 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.129:33615
--local_ip_for_outbound_sockets=127.27.17.129
--tserver_master_addrs=127.27.17.190:33447
--webserver_port=35231
--webserver_interface=127.27.17.129
--env_inject_eio_globs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-1/**
--env_inject_eio=1.0
--crash_on_eio=false
--builtin_ntp_servers=127.27.17.148:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20260810 22:05:32.539023 3294 connection.cc:535] client connection to 127.27.17.129:33615 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 22:05:32.539494 3294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 22:05:32.547894 3294 consensus_peers.cc:487] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a -> Peer 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): Couldn't send request to peer 742853e1f3ae4ac1b86265b6adb193f9. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:33615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 22:05:32.557140 3907 consensus_queue.cc:1035] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 22:05:32.574359 3294 consensus_peers.cc:487] T 159f82c64cc8402fb2570de1b5223e78 P bdb9b70cb73a412ab0d4049695126b0a -> Peer 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): Couldn't send request to peer 742853e1f3ae4ac1b86265b6adb193f9. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:33615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 22:05:32.870026 3910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:05:32.870779 3910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:05:32.871570 3910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:05:32.903923 3910 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20260810 22:05:32.904345 3910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:05:32.904949 3910 flags.cc:425] Enabled experimental flag: --crash_on_eio=false
W20260810 22:05:32.905530 3910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.129
I20260810 22:05:32.941833 3910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.129:33615
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.27.17.129
--webserver_port=35231
--tserver_master_addrs=127.27.17.190:33447
--env_inject_eio=1
--env_inject_eio_globs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-1/**
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.129
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:05:32.943125 3910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260810 22:05:32.944186 3910 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260810 22:05:32.944378 3910 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260810 22:05:32.945619 3910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:05:32.966882 3924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:33.490751 3829 raft_consensus.cc:482] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:33.491271 3829 raft_consensus.cc:504] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:33.493289 3829 leader_election.cc:288] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers da5bd62264364fd198587af606253dd1 (127.27.17.132:41969), 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615)
I20260810 22:05:33.494764 3690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943ec2cbc29e44e3bcbe0e1e5b039f55" candidate_uuid: "bdb9b70cb73a412ab0d4049695126b0a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da5bd62264364fd198587af606253dd1" is_pre_election: true
I20260810 22:05:33.495705 3690 raft_consensus.cc:2454] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdb9b70cb73a412ab0d4049695126b0a in term 1.
I20260810 22:05:33.497644 3292 leader_election.cc:302] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdb9b70cb73a412ab0d4049695126b0a, da5bd62264364fd198587af606253dd1; no voters:
I20260810 22:05:33.498941 3829 raft_consensus.cc:2781] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260810 22:05:33.499248 3294 leader_election.cc:334] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): Network error: Client connection negotiation failed: client connection to 127.27.17.129:33615: connect: Connection refused (error 111)
I20260810 22:05:33.499506 3829 raft_consensus.cc:482] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:33.500288 3829 raft_consensus.cc:3037] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:05:33.510578 3829 raft_consensus.cc:504] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:33.513314 3829 leader_election.cc:288] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [CANDIDATE]: Term 2 election: Requested vote from peers da5bd62264364fd198587af606253dd1 (127.27.17.132:41969), 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615)
I20260810 22:05:33.514189 3690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943ec2cbc29e44e3bcbe0e1e5b039f55" candidate_uuid: "bdb9b70cb73a412ab0d4049695126b0a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da5bd62264364fd198587af606253dd1"
I20260810 22:05:33.514794 3690 raft_consensus.cc:3037] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 1 FOLLOWER]: Advancing to term 2
W20260810 22:05:33.520218 3294 leader_election.cc:334] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): Network error: Client connection negotiation failed: client connection to 127.27.17.129:33615: connect: Connection refused (error 111)
I20260810 22:05:33.522071 3690 raft_consensus.cc:2454] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdb9b70cb73a412ab0d4049695126b0a in term 2.
I20260810 22:05:33.523161 3292 leader_election.cc:302] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bdb9b70cb73a412ab0d4049695126b0a, da5bd62264364fd198587af606253dd1; no voters: 742853e1f3ae4ac1b86265b6adb193f9
I20260810 22:05:33.524042 3829 raft_consensus.cc:2781] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:05:33.525878 3829 raft_consensus.cc:686] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [term 2 LEADER]: Becoming Leader. State: Replica: bdb9b70cb73a412ab0d4049695126b0a, State: Running, Role: LEADER
I20260810 22:05:33.527101 3829 consensus_queue.cc:237] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:33.550410 2473 catalog_manager.cc:5495] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a reported cstate change: term changed from 1 to 2, leader changed from 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129) to bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131). New cstate: current_term: 2 leader_uuid: "bdb9b70cb73a412ab0d4049695126b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } health_report { overall_health: UNKNOWN } } }
I20260810 22:05:33.589643 3940 raft_consensus.cc:482] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:33.590093 3940 raft_consensus.cc:504] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:33.591877 3940 leader_election.cc:288] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers da5bd62264364fd198587af606253dd1 (127.27.17.132:41969), 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615)
W20260810 22:05:33.607161 3013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.129:33615: connect: Connection refused (error 111)
W20260810 22:05:33.611702 3013 leader_election.cc:334] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): Network error: Client connection negotiation failed: client connection to 127.27.17.129:33615: connect: Connection refused (error 111)
I20260810 22:05:33.618105 3690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "391eba1c39404071969e21ddd5abef69" candidate_uuid: "53b64278bdc549e9a56784b6b3295490" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da5bd62264364fd198587af606253dd1" is_pre_election: true
I20260810 22:05:33.618665 3690 raft_consensus.cc:2454] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53b64278bdc549e9a56784b6b3295490 in term 1.
I20260810 22:05:33.619747 3010 leader_election.cc:302] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53b64278bdc549e9a56784b6b3295490, da5bd62264364fd198587af606253dd1; no voters: 742853e1f3ae4ac1b86265b6adb193f9
I20260810 22:05:33.620857 3940 raft_consensus.cc:2781] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 22:05:33.621184 3940 raft_consensus.cc:482] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:33.621565 3940 raft_consensus.cc:3037] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:05:33.626241 3940 raft_consensus.cc:504] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:33.627782 3940 leader_election.cc:288] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [CANDIDATE]: Term 2 election: Requested vote from peers da5bd62264364fd198587af606253dd1 (127.27.17.132:41969), 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615)
I20260810 22:05:33.629207 3690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "391eba1c39404071969e21ddd5abef69" candidate_uuid: "53b64278bdc549e9a56784b6b3295490" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da5bd62264364fd198587af606253dd1"
I20260810 22:05:33.630100 3690 raft_consensus.cc:3037] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1 [term 1 FOLLOWER]: Advancing to term 2
W20260810 22:05:33.631536 3013 leader_election.cc:334] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): Network error: Client connection negotiation failed: client connection to 127.27.17.129:33615: connect: Connection refused (error 111)
I20260810 22:05:33.639910 3690 raft_consensus.cc:2454] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53b64278bdc549e9a56784b6b3295490 in term 2.
I20260810 22:05:33.641399 3010 leader_election.cc:302] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53b64278bdc549e9a56784b6b3295490, da5bd62264364fd198587af606253dd1; no voters: 742853e1f3ae4ac1b86265b6adb193f9
I20260810 22:05:33.642117 3940 raft_consensus.cc:2781] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:05:33.643514 3940 raft_consensus.cc:686] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [term 2 LEADER]: Becoming Leader. State: Replica: 53b64278bdc549e9a56784b6b3295490, State: Running, Role: LEADER
I20260810 22:05:33.644570 3940 consensus_queue.cc:237] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } }
I20260810 22:05:33.655376 2473 catalog_manager.cc:5495] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 reported cstate change: term changed from 1 to 2, leader changed from 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129) to 53b64278bdc549e9a56784b6b3295490 (127.27.17.130). New cstate: current_term: 2 leader_uuid: "53b64278bdc549e9a56784b6b3295490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } health_report { overall_health: UNKNOWN } } }
I20260810 22:05:33.972088 3690 raft_consensus.cc:1261] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 2 FOLLOWER]: Refusing update from remote peer bdb9b70cb73a412ab0d4049695126b0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260810 22:05:33.974201 3829 consensus_queue.cc:1035] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 22:05:33.999238 3294 consensus_peers.cc:487] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a -> Peer 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): Couldn't send request to peer 742853e1f3ae4ac1b86265b6adb193f9. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:33615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 22:05:34.093421 3690 raft_consensus.cc:1261] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1 [term 2 FOLLOWER]: Refusing update from remote peer 53b64278bdc549e9a56784b6b3295490: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260810 22:05:34.094702 3940 consensus_queue.cc:1035] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 22:05:34.114403 3013 consensus_peers.cc:487] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 -> Peer 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): Couldn't send request to peer 742853e1f3ae4ac1b86265b6adb193f9. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:33615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 22:05:32.967675 3926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:32.968590 3910 server_base.cc:1034] running on GCE node
W20260810 22:05:32.968158 3923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:34.227762 3910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:05:34.230451 3910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:05:34.231889 3910 hybrid_clock.cc:648] HybridClock initialized: now 1786399534231857 us; error 45 us; skew 500 ppm
I20260810 22:05:34.232860 3910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:05:34.239850 3910 webserver.cc:460] Webserver started at http://127.27.17.129:35231/ using document root <none> and password file <none>
I20260810 22:05:34.241693 3910 fs_manager.cc:359] Metadata directory not provided
I20260810 22:05:34.241958 3910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
E20260810 22:05:34.244535 3910 env_posix.cc:432] I/O error, context: INJECTED FAILURE
E20260810 22:05:34.249704 3910 env_posix.cc:432] I/O error, context: INJECTED FAILURE
E20260810 22:05:34.265702 3910 env_posix.cc:432] I/O error, context: INJECTED FAILURE
I20260810 22:05:34.277964 3910 fs_manager.cc:709] Time spent opening directory manager: real 0.030s user 0.024s sys 0.007s
E20260810 22:05:34.683120 3910 log_block_manager.cc:2924] Not using report from /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-1/data: IO error: Data directory failed (error 5)
I20260810 22:05:34.685736 3910 fs_manager.cc:725] Time spent opening block manager: real 0.405s user 0.004s sys 0.001s
I20260810 22:05:34.686098 3910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/wal
uuid: "742853e1f3ae4ac1b86265b6adb193f9"
format_stamp: "Formatted at 2026-08-10 22:05:22 on dist-test-slave-bw88"
I20260810 22:05:34.686635 3910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/wal
4 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:05:34.743219 3910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:05:34.744850 3910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:05:34.745429 3910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:05:34.747589 3910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:05:34.753541 4078 ts_tablet_manager.cc:542] Loading tablet metadata (0/4 complete)
W20260810 22:05:34.765605 4082 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet cfb52501fc724335a0c3d3e6a7ee95db: could not find data dir with uuid c5c24efc673a416cb0c8ec83083610b7
W20260810 22:05:34.765661 4078 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet 391eba1c39404071969e21ddd5abef69: could not find data dir with uuid c5c24efc673a416cb0c8ec83083610b7
W20260810 22:05:34.765983 4079 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet 943ec2cbc29e44e3bcbe0e1e5b039f55: could not find data dir with uuid c5c24efc673a416cb0c8ec83083610b7
I20260810 22:05:34.770551 3910 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20260810 22:05:34.770790 3910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.019s user 0.001s sys 0.011s
I20260810 22:05:34.771070 3910 ts_tablet_manager.cc:594] Registering tablets (0/4 complete)
I20260810 22:05:34.776810 4083 tablet_bootstrap.cc:492] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9: Bootstrap starting.
E20260810 22:05:34.777279 4083 ts_tablet_manager.cc:1419] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 391eba1c39404071969e21ddd5abef69
I20260810 22:05:34.777590 4083 ts_tablet_manager.cc:1397] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:05:34.777853 4083 tablet_replica.cc:330] stopping tablet replica
I20260810 22:05:34.780567 4079 tablet_bootstrap.cc:492] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9: Bootstrap starting.
I20260810 22:05:34.778138 4083 raft_consensus.cc:2229] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:05:34.780980 4083 raft_consensus.cc:2258] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 FOLLOWER]: Raft consensus is shut down!
E20260810 22:05:34.780980 4079 ts_tablet_manager.cc:1419] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 943ec2cbc29e44e3bcbe0e1e5b039f55
I20260810 22:05:34.781401 4079 ts_tablet_manager.cc:1397] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:05:34.781651 4079 tablet_replica.cc:330] stopping tablet replica
I20260810 22:05:34.781880 4079 raft_consensus.cc:2229] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:05:34.782153 4079 raft_consensus.cc:2258] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:05:34.784490 4079 tablet_bootstrap.cc:492] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9: Bootstrap starting.
E20260810 22:05:34.784833 4079 ts_tablet_manager.cc:1419] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet cfb52501fc724335a0c3d3e6a7ee95db
I20260810 22:05:34.785048 4079 ts_tablet_manager.cc:1397] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:05:34.785259 4079 tablet_replica.cc:330] stopping tablet replica
I20260810 22:05:34.785528 4079 raft_consensus.cc:2229] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:05:34.785795 4079 raft_consensus.cc:2258] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:05:34.787998 3910 ts_tablet_manager.cc:610] Registered 4 tablets
I20260810 22:05:34.788218 3910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.017s user 0.014s sys 0.001s
I20260810 22:05:34.788280 4079 tablet_bootstrap.cc:492] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9: Bootstrap starting.
I20260810 22:05:34.842768 4079 log.cc:824] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9: Log is configured to *not* fsync() on all Append() calls
I20260810 22:05:34.852034 4079 tablet_bootstrap.cc:492] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:05:34.853193 4079 tablet_bootstrap.cc:492] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9: Bootstrap complete.
I20260810 22:05:34.854938 4079 ts_tablet_manager.cc:1397] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9: Time spent bootstrapping tablet: real 0.067s user 0.027s sys 0.035s
I20260810 22:05:34.888386 4079 raft_consensus.cc:348] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } }
I20260810 22:05:34.889902 4079 raft_consensus.cc:729] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742853e1f3ae4ac1b86265b6adb193f9, State: Initialized, Role: FOLLOWER
I20260810 22:05:34.891387 4079 consensus_queue.cc:260] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "742853e1f3ae4ac1b86265b6adb193f9" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 33615 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } }
I20260810 22:05:34.900730 4079 ts_tablet_manager.cc:1428] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9: Time spent starting tablet: real 0.046s user 0.032s sys 0.009s
W20260810 22:05:34.966368 3294 consensus_peers.cc:487] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a -> Peer 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): Couldn't send request to peer 742853e1f3ae4ac1b86265b6adb193f9. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.129:33615: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260810 22:05:34.994894 3910 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.129:33615
I20260810 22:05:34.995148 4233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.129:33615 every 8 connection(s)
I20260810 22:05:34.999222 3910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
W20260810 22:05:35.004884 4235 tablet.cc:2369] T 159f82c64cc8402fb2570de1b5223e78 P 742853e1f3ae4ac1b86265b6adb193f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:05:35.010015 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 3910
I20260810 22:05:35.042953 4234 heartbeater.cc:347] Connected to a master server at 127.27.17.190:33447
I20260810 22:05:35.043447 4234 heartbeater.cc:464] Registering TS with master...
I20260810 22:05:35.044759 4234 heartbeater.cc:511] Master 127.27.17.190:33447 requested a full tablet report, sending...
I20260810 22:05:35.050079 2473 ts_manager.cc:194] Re-registered known tserver with Master: 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615)
W20260810 22:05:35.052819 2473 catalog_manager.cc:5408] Tablet 391eba1c39404071969e21ddd5abef69 (table test-workload [id=3cc1488e248b4e99a0705e301f54d797]) has failed on TS 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 391eba1c39404071969e21ddd5abef69
W20260810 22:05:35.053366 2473 catalog_manager.cc:5408] Tablet 943ec2cbc29e44e3bcbe0e1e5b039f55 (table test-workload [id=3cc1488e248b4e99a0705e301f54d797]) has failed on TS 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 943ec2cbc29e44e3bcbe0e1e5b039f55
W20260810 22:05:35.053794 2473 catalog_manager.cc:5408] Tablet cfb52501fc724335a0c3d3e6a7ee95db (table test-workload [id=3cc1488e248b4e99a0705e301f54d797]) has failed on TS 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet cfb52501fc724335a0c3d3e6a7ee95db
I20260810 22:05:35.058296 2473 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.129:38957
I20260810 22:05:35.179085 3013 consensus_queue.cc:579] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [LEADER]: The tablet replica hosted on peer 742853e1f3ae4ac1b86265b6adb193f9 has failed
I20260810 22:05:35.192219 3098 consensus_queue.cc:237] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } }
I20260810 22:05:35.199985 4234 heartbeater.cc:503] Master 127.27.17.190:33447 was elected leader, sending a full tablet report...
I20260810 22:05:35.213672 3690 raft_consensus.cc:1261] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1 [term 2 FOLLOWER]: Refusing update from remote peer 53b64278bdc549e9a56784b6b3295490: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 22:05:35.215158 4059 consensus_queue.cc:1035] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260810 22:05:35.216852 2473 catalog_manager.cc:5408] Tablet 391eba1c39404071969e21ddd5abef69 (table test-workload [id=3cc1488e248b4e99a0705e301f54d797]) has failed on TS 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 391eba1c39404071969e21ddd5abef69
W20260810 22:05:35.217622 2473 catalog_manager.cc:5408] Tablet 943ec2cbc29e44e3bcbe0e1e5b039f55 (table test-workload [id=3cc1488e248b4e99a0705e301f54d797]) has failed on TS 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 943ec2cbc29e44e3bcbe0e1e5b039f55
W20260810 22:05:35.218220 2473 catalog_manager.cc:5408] Tablet cfb52501fc724335a0c3d3e6a7ee95db (table test-workload [id=3cc1488e248b4e99a0705e301f54d797]) has failed on TS 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet cfb52501fc724335a0c3d3e6a7ee95db
I20260810 22:05:35.224735 4255 raft_consensus.cc:2932] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } }
I20260810 22:05:35.226295 3690 raft_consensus.cc:2932] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } }
I20260810 22:05:35.234709 2456 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 391eba1c39404071969e21ddd5abef69 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 22:05:35.238639 2473 catalog_manager.cc:5495] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1 reported cstate change: config changed from index -1 to 2, VOTER 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129) evicted. New cstate: current_term: 2 leader_uuid: "53b64278bdc549e9a56784b6b3295490" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } }
I20260810 22:05:35.258777 3098 consensus_queue.cc:237] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: NON_VOTER last_known_addr { host: "127.27.17.131" port: 45861 } attrs { promote: true } }
I20260810 22:05:35.266959 3690 raft_consensus.cc:1261] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1 [term 2 FOLLOWER]: Refusing update from remote peer 53b64278bdc549e9a56784b6b3295490: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260810 22:05:35.267859 3013 consensus_peers.cc:487] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 -> Peer bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131:45861): Couldn't send request to peer bdb9b70cb73a412ab0d4049695126b0a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 391eba1c39404071969e21ddd5abef69. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:05:35.268622 4263 consensus_queue.cc:1035] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 22:05:35.269563 4147 tablet_service.cc:1514] Processing DeleteTablet for tablet 391eba1c39404071969e21ddd5abef69 with delete_type TABLET_DATA_TOMBSTONED (TS 742853e1f3ae4ac1b86265b6adb193f9 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:47144
I20260810 22:05:35.271425 4270 ts_tablet_manager.cc:1905] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 22:05:35.274466 4059 raft_consensus.cc:2932] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, NON_VOTER bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: NON_VOTER last_known_addr { host: "127.27.17.131" port: 45861 } attrs { promote: true } } }
I20260810 22:05:35.277808 3690 raft_consensus.cc:2932] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, NON_VOTER bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: NON_VOTER last_known_addr { host: "127.27.17.131" port: 45861 } attrs { promote: true } } }
I20260810 22:05:35.281987 2456 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 391eba1c39404071969e21ddd5abef69 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260810 22:05:35.287580 2475 catalog_manager.cc:5495] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 reported cstate change: config changed from index 2 to 3, NON_VOTER bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131) added. New cstate: current_term: 2 leader_uuid: "53b64278bdc549e9a56784b6b3295490" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: NON_VOTER last_known_addr { host: "127.27.17.131" port: 45861 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260810 22:05:35.296279 4270 ts_tablet_manager.cc:1918] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260810 22:05:35.296648 4270 log.cc:1196] T 391eba1c39404071969e21ddd5abef69 P 742853e1f3ae4ac1b86265b6adb193f9: Deleting WAL directory at /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/wal/wals/391eba1c39404071969e21ddd5abef69
I20260810 22:05:35.298614 2456 catalog_manager.cc:4841] TS 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): tablet 391eba1c39404071969e21ddd5abef69 (table test-workload [id=3cc1488e248b4e99a0705e301f54d797]) successfully deleted
I20260810 22:05:35.406699 3067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 22:05:35.420051 3355 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 22:05:35.425127 3658 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 22:05:35.438349 3294 consensus_queue.cc:579] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: The tablet replica hosted on peer 742853e1f3ae4ac1b86265b6adb193f9 has failed
I20260810 22:05:35.452250 4147 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 22:05:35.471364 3384 consensus_queue.cc:237] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } }
I20260810 22:05:35.484468 3098 raft_consensus.cc:1261] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490 [term 1 FOLLOWER]: Refusing update from remote peer bdb9b70cb73a412ab0d4049695126b0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:05:35.486202 4304 consensus_queue.cc:1035] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 22:05:35.488530 3294 consensus_queue.cc:579] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: The tablet replica hosted on peer 742853e1f3ae4ac1b86265b6adb193f9 has failed
I20260810 22:05:35.499058 3384 consensus_queue.cc:237] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } }
I20260810 22:05:35.507587 4304 raft_consensus.cc:2932] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } }
I20260810 22:05:35.510080 3098 raft_consensus.cc:2932] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } }
I20260810 22:05:35.517684 2456 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet cfb52501fc724335a0c3d3e6a7ee95db with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 22:05:35.520664 3690 raft_consensus.cc:1261] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 2 FOLLOWER]: Refusing update from remote peer bdb9b70cb73a412ab0d4049695126b0a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 22:05:35.522223 3953 consensus_queue.cc:1035] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 22:05:35.529537 2473 catalog_manager.cc:5495] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a reported cstate change: config changed from index -1 to 2, VOTER 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129) evicted. New cstate: current_term: 1 leader_uuid: "bdb9b70cb73a412ab0d4049695126b0a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } health_report { overall_health: HEALTHY } } }
I20260810 22:05:35.535517 3953 raft_consensus.cc:2932] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } }
I20260810 22:05:35.544710 3384 consensus_queue.cc:237] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: NON_VOTER last_known_addr { host: "127.27.17.132" port: 41969 } attrs { promote: true } }
I20260810 22:05:35.546991 3690 raft_consensus.cc:2932] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } }
I20260810 22:05:35.550625 2456 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 943ec2cbc29e44e3bcbe0e1e5b039f55 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 22:05:35.551542 4147 tablet_service.cc:1514] Processing DeleteTablet for tablet cfb52501fc724335a0c3d3e6a7ee95db with delete_type TABLET_DATA_TOMBSTONED (TS 742853e1f3ae4ac1b86265b6adb193f9 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:47144
I20260810 22:05:35.552536 4270 ts_tablet_manager.cc:1905] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260810 22:05:35.562867 3292 consensus_peers.cc:487] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a -> Peer da5bd62264364fd198587af606253dd1 (127.27.17.132:41969): Couldn't send request to peer da5bd62264364fd198587af606253dd1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cfb52501fc724335a0c3d3e6a7ee95db. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:05:35.567193 3099 raft_consensus.cc:1261] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490 [term 1 FOLLOWER]: Refusing update from remote peer bdb9b70cb73a412ab0d4049695126b0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 22:05:35.566628 2475 catalog_manager.cc:5495] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 reported cstate change: config changed from index -1 to 2, VOTER 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129) evicted. New cstate: current_term: 2 leader_uuid: "bdb9b70cb73a412ab0d4049695126b0a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } }
I20260810 22:05:35.573077 4304 consensus_queue.cc:1035] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 22:05:35.580355 4301 raft_consensus.cc:2932] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER da5bd62264364fd198587af606253dd1 (127.27.17.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: NON_VOTER last_known_addr { host: "127.27.17.132" port: 41969 } attrs { promote: true } } }
I20260810 22:05:35.581219 4147 tablet_service.cc:1514] Processing DeleteTablet for tablet 943ec2cbc29e44e3bcbe0e1e5b039f55 with delete_type TABLET_DATA_TOMBSTONED (TS 742853e1f3ae4ac1b86265b6adb193f9 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:47144
I20260810 22:05:35.585469 3384 consensus_queue.cc:237] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: NON_VOTER last_known_addr { host: "127.27.17.130" port: 44557 } attrs { promote: true } }
I20260810 22:05:35.587014 3098 raft_consensus.cc:2932] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER da5bd62264364fd198587af606253dd1 (127.27.17.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: NON_VOTER last_known_addr { host: "127.27.17.132" port: 41969 } attrs { promote: true } } }
W20260810 22:05:35.591830 3294 consensus_peers.cc:487] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a -> Peer 53b64278bdc549e9a56784b6b3295490 (127.27.17.130:44557): Couldn't send request to peer 53b64278bdc549e9a56784b6b3295490. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 943ec2cbc29e44e3bcbe0e1e5b039f55. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:05:35.591886 3690 raft_consensus.cc:1261] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 2 FOLLOWER]: Refusing update from remote peer bdb9b70cb73a412ab0d4049695126b0a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260810 22:05:35.593861 4304 consensus_queue.cc:1035] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 22:05:35.601150 3953 raft_consensus.cc:2932] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, NON_VOTER 53b64278bdc549e9a56784b6b3295490 (127.27.17.130) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: NON_VOTER last_known_addr { host: "127.27.17.130" port: 44557 } attrs { promote: true } } }
I20260810 22:05:35.603111 3690 raft_consensus.cc:2932] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, NON_VOTER 53b64278bdc549e9a56784b6b3295490 (127.27.17.130) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: NON_VOTER last_known_addr { host: "127.27.17.130" port: 44557 } attrs { promote: true } } }
I20260810 22:05:35.607883 4310 ts_tablet_manager.cc:1905] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 22:05:35.618779 2456 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 943ec2cbc29e44e3bcbe0e1e5b039f55 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260810 22:05:35.620826 2456 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cfb52501fc724335a0c3d3e6a7ee95db with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260810 22:05:35.626909 4270 ts_tablet_manager.cc:1918] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260810 22:05:35.627318 4270 log.cc:1196] T cfb52501fc724335a0c3d3e6a7ee95db P 742853e1f3ae4ac1b86265b6adb193f9: Deleting WAL directory at /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/wal/wals/cfb52501fc724335a0c3d3e6a7ee95db
I20260810 22:05:35.628919 2456 catalog_manager.cc:4841] TS 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): tablet cfb52501fc724335a0c3d3e6a7ee95db (table test-workload [id=3cc1488e248b4e99a0705e301f54d797]) successfully deleted
I20260810 22:05:35.628983 2473 catalog_manager.cc:5495] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490 reported cstate change: config changed from index 2 to 3, NON_VOTER da5bd62264364fd198587af606253dd1 (127.27.17.132) added. New cstate: current_term: 1 leader_uuid: "bdb9b70cb73a412ab0d4049695126b0a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: NON_VOTER last_known_addr { host: "127.27.17.132" port: 41969 } attrs { promote: true } } }
I20260810 22:05:35.627918 2474 catalog_manager.cc:5495] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 reported cstate change: config changed from index 2 to 3, NON_VOTER 53b64278bdc549e9a56784b6b3295490 (127.27.17.130) added. New cstate: current_term: 2 leader_uuid: "bdb9b70cb73a412ab0d4049695126b0a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: NON_VOTER last_known_addr { host: "127.27.17.130" port: 44557 } attrs { promote: true } } }
I20260810 22:05:35.671461 4310 ts_tablet_manager.cc:1918] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260810 22:05:35.671893 4310 log.cc:1196] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 742853e1f3ae4ac1b86265b6adb193f9: Deleting WAL directory at /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/wal/wals/943ec2cbc29e44e3bcbe0e1e5b039f55
I20260810 22:05:35.673589 2456 catalog_manager.cc:4841] TS 742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): tablet 943ec2cbc29e44e3bcbe0e1e5b039f55 (table test-workload [id=3cc1488e248b4e99a0705e301f54d797]) successfully deleted
I20260810 22:05:35.733983 4311 ts_tablet_manager.cc:927] T 391eba1c39404071969e21ddd5abef69 P bdb9b70cb73a412ab0d4049695126b0a: Initiating tablet copy from peer 53b64278bdc549e9a56784b6b3295490 (127.27.17.130:44557)
I20260810 22:05:35.739845 4311 tablet_copy_client.cc:316] T 391eba1c39404071969e21ddd5abef69 P bdb9b70cb73a412ab0d4049695126b0a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.17.130:44557
I20260810 22:05:35.742007 3127 tablet_copy_service.cc:140] P 53b64278bdc549e9a56784b6b3295490: Received BeginTabletCopySession request for tablet 391eba1c39404071969e21ddd5abef69 from peer bdb9b70cb73a412ab0d4049695126b0a ({username='slave'} at 127.27.17.131:57635)
I20260810 22:05:35.742568 3127 tablet_copy_service.cc:161] P 53b64278bdc549e9a56784b6b3295490: Beginning new tablet copy session on tablet 391eba1c39404071969e21ddd5abef69 from peer bdb9b70cb73a412ab0d4049695126b0a at {username='slave'} at 127.27.17.131:57635: session id = bdb9b70cb73a412ab0d4049695126b0a-391eba1c39404071969e21ddd5abef69
I20260810 22:05:35.750844 3127 tablet_copy_source_session.cc:205] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:05:35.782393 4311 tablet_copy_client.cc:791] T 391eba1c39404071969e21ddd5abef69 P bdb9b70cb73a412ab0d4049695126b0a: tablet copy: Starting download of 0 data blocks...
I20260810 22:05:35.782994 4311 tablet_copy_client.cc:655] T 391eba1c39404071969e21ddd5abef69 P bdb9b70cb73a412ab0d4049695126b0a: tablet copy: Starting download of 1 WAL segments...
I20260810 22:05:35.797545 4311 tablet_copy_client.cc:531] T 391eba1c39404071969e21ddd5abef69 P bdb9b70cb73a412ab0d4049695126b0a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:05:35.810113 4311 tablet_bootstrap.cc:492] T 391eba1c39404071969e21ddd5abef69 P bdb9b70cb73a412ab0d4049695126b0a: Bootstrap starting.
I20260810 22:05:35.867189 4311 tablet_bootstrap.cc:492] T 391eba1c39404071969e21ddd5abef69 P bdb9b70cb73a412ab0d4049695126b0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:05:35.868048 4311 tablet_bootstrap.cc:492] T 391eba1c39404071969e21ddd5abef69 P bdb9b70cb73a412ab0d4049695126b0a: Bootstrap complete.
I20260810 22:05:35.868680 4311 ts_tablet_manager.cc:1397] T 391eba1c39404071969e21ddd5abef69 P bdb9b70cb73a412ab0d4049695126b0a: Time spent bootstrapping tablet: real 0.059s user 0.020s sys 0.005s
I20260810 22:05:35.871701 4311 raft_consensus.cc:348] T 391eba1c39404071969e21ddd5abef69 P bdb9b70cb73a412ab0d4049695126b0a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: NON_VOTER last_known_addr { host: "127.27.17.131" port: 45861 } attrs { promote: true } }
I20260810 22:05:35.872460 4311 raft_consensus.cc:729] T 391eba1c39404071969e21ddd5abef69 P bdb9b70cb73a412ab0d4049695126b0a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: bdb9b70cb73a412ab0d4049695126b0a, State: Initialized, Role: LEARNER
I20260810 22:05:35.872998 4311 consensus_queue.cc:260] T 391eba1c39404071969e21ddd5abef69 P bdb9b70cb73a412ab0d4049695126b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: NON_VOTER last_known_addr { host: "127.27.17.131" port: 45861 } attrs { promote: true } }
I20260810 22:05:35.875067 4311 ts_tablet_manager.cc:1428] T 391eba1c39404071969e21ddd5abef69 P bdb9b70cb73a412ab0d4049695126b0a: Time spent starting tablet: real 0.006s user 0.005s sys 0.003s
I20260810 22:05:35.876708 3127 tablet_copy_service.cc:342] P 53b64278bdc549e9a56784b6b3295490: Request end of tablet copy session bdb9b70cb73a412ab0d4049695126b0a-391eba1c39404071969e21ddd5abef69 received from {username='slave'} at 127.27.17.131:57635
I20260810 22:05:35.877090 3127 tablet_copy_service.cc:433] P 53b64278bdc549e9a56784b6b3295490: ending tablet copy session bdb9b70cb73a412ab0d4049695126b0a-391eba1c39404071969e21ddd5abef69 on tablet 391eba1c39404071969e21ddd5abef69 with peer bdb9b70cb73a412ab0d4049695126b0a
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
12ce8059216a4cb68b7ba53cac315b54 | 127.27.17.190:33447 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:37169 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
53b64278bdc549e9a56784b6b3295490 | 127.27.17.130:44557 | HEALTHY | <none> | 1 | 0
742853e1f3ae4ac1b86265b6adb193f9 | 127.27.17.129:33615 | HEALTHY | <none> | 0 | 0
bdb9b70cb73a412ab0d4049695126b0a | 127.27.17.131:45861 | HEALTHY | <none> | 3 | 0
da5bd62264364fd198587af606253dd1 | 127.27.17.132:41969 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.27.17.129:33615
env_inject_eio | 1 | hidden | 127.27.17.129:33615
env_inject_eio_globs | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-1/** | hidden | 127.27.17.129:33615
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.27.17.129 | experimental | 127.27.17.129:33615
local_ip_for_outbound_sockets | 127.27.17.130 | experimental | 127.27.17.130:44557
local_ip_for_outbound_sockets | 127.27.17.131 | experimental | 127.27.17.131:45861
local_ip_for_outbound_sockets | 127.27.17.132 | experimental | 127.27.17.132:41969
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.27.17.129:33615
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.27.17.130:44557
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.27.17.131:45861
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.27.17.132:41969
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:37169 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet cfb52501fc724335a0c3d3e6a7ee95db of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131:45861): RUNNING [LEADER]
742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet cfb52501fc724335a0c3d3e6a7ee95db
53b64278bdc549e9a56784b6b3295490 (127.27.17.130:44557): RUNNING
All reported replicas are:
A = bdb9b70cb73a412ab0d4049695126b0a
B = 742853e1f3ae4ac1b86265b6adb193f9
C = 53b64278bdc549e9a56784b6b3295490
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 943ec2cbc29e44e3bcbe0e1e5b039f55 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131:45861): RUNNING [LEADER]
da5bd62264364fd198587af606253dd1 (127.27.17.132:41969): RUNNING
742853e1f3ae4ac1b86265b6adb193f9 (127.27.17.129:33615): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 943ec2cbc29e44e3bcbe0e1e5b039f55
All reported replicas are:
A = bdb9b70cb73a412ab0d4049695126b0a
B = da5bd62264364fd198587af606253dd1
C = 742853e1f3ae4ac1b86265b6adb193f9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | A* B C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 5 | 3 | 0 | 2 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 4
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 22:05:35.923162 27718 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 22:05:35.966478 4315 ts_tablet_manager.cc:927] T cfb52501fc724335a0c3d3e6a7ee95db P da5bd62264364fd198587af606253dd1: Initiating tablet copy from peer bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131:45861)
I20260810 22:05:35.968134 4315 tablet_copy_client.cc:316] T cfb52501fc724335a0c3d3e6a7ee95db P da5bd62264364fd198587af606253dd1: tablet copy: Beginning tablet copy session from remote peer at address 127.27.17.131:45861
I20260810 22:05:35.969851 3412 tablet_copy_service.cc:140] P bdb9b70cb73a412ab0d4049695126b0a: Received BeginTabletCopySession request for tablet cfb52501fc724335a0c3d3e6a7ee95db from peer da5bd62264364fd198587af606253dd1 ({username='slave'} at 127.27.17.132:58877)
I20260810 22:05:35.970278 3412 tablet_copy_service.cc:161] P bdb9b70cb73a412ab0d4049695126b0a: Beginning new tablet copy session on tablet cfb52501fc724335a0c3d3e6a7ee95db from peer da5bd62264364fd198587af606253dd1 at {username='slave'} at 127.27.17.132:58877: session id = da5bd62264364fd198587af606253dd1-cfb52501fc724335a0c3d3e6a7ee95db
I20260810 22:05:35.975683 3412 tablet_copy_source_session.cc:205] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:05:35.988755 4315 tablet_copy_client.cc:791] T cfb52501fc724335a0c3d3e6a7ee95db P da5bd62264364fd198587af606253dd1: tablet copy: Starting download of 0 data blocks...
I20260810 22:05:35.989319 4315 tablet_copy_client.cc:655] T cfb52501fc724335a0c3d3e6a7ee95db P da5bd62264364fd198587af606253dd1: tablet copy: Starting download of 1 WAL segments...
I20260810 22:05:35.993175 4315 tablet_copy_client.cc:531] T cfb52501fc724335a0c3d3e6a7ee95db P da5bd62264364fd198587af606253dd1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:05:35.998871 4315 tablet_bootstrap.cc:492] T cfb52501fc724335a0c3d3e6a7ee95db P da5bd62264364fd198587af606253dd1: Bootstrap starting.
I20260810 22:05:36.017447 4315 tablet_bootstrap.cc:492] T cfb52501fc724335a0c3d3e6a7ee95db P da5bd62264364fd198587af606253dd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:05:36.018327 4315 tablet_bootstrap.cc:492] T cfb52501fc724335a0c3d3e6a7ee95db P da5bd62264364fd198587af606253dd1: Bootstrap complete.
I20260810 22:05:36.018944 4315 ts_tablet_manager.cc:1397] T cfb52501fc724335a0c3d3e6a7ee95db P da5bd62264364fd198587af606253dd1: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.004s
I20260810 22:05:36.020839 4315 raft_consensus.cc:348] T cfb52501fc724335a0c3d3e6a7ee95db P da5bd62264364fd198587af606253dd1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: NON_VOTER last_known_addr { host: "127.27.17.132" port: 41969 } attrs { promote: true } }
I20260810 22:05:36.021442 4315 raft_consensus.cc:729] T cfb52501fc724335a0c3d3e6a7ee95db P da5bd62264364fd198587af606253dd1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da5bd62264364fd198587af606253dd1, State: Initialized, Role: LEARNER
I20260810 22:05:36.021983 4315 consensus_queue.cc:260] T cfb52501fc724335a0c3d3e6a7ee95db P da5bd62264364fd198587af606253dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: NON_VOTER last_known_addr { host: "127.27.17.132" port: 41969 } attrs { promote: true } }
I20260810 22:05:36.023967 4315 ts_tablet_manager.cc:1428] T cfb52501fc724335a0c3d3e6a7ee95db P da5bd62264364fd198587af606253dd1: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:05:36.025794 3412 tablet_copy_service.cc:342] P bdb9b70cb73a412ab0d4049695126b0a: Request end of tablet copy session da5bd62264364fd198587af606253dd1-cfb52501fc724335a0c3d3e6a7ee95db received from {username='slave'} at 127.27.17.132:58877
I20260810 22:05:36.026276 3412 tablet_copy_service.cc:433] P bdb9b70cb73a412ab0d4049695126b0a: ending tablet copy session da5bd62264364fd198587af606253dd1-cfb52501fc724335a0c3d3e6a7ee95db on tablet cfb52501fc724335a0c3d3e6a7ee95db with peer da5bd62264364fd198587af606253dd1
I20260810 22:05:36.057544 4320 ts_tablet_manager.cc:927] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 53b64278bdc549e9a56784b6b3295490: Initiating tablet copy from peer bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131:45861)
I20260810 22:05:36.060654 4320 tablet_copy_client.cc:316] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 53b64278bdc549e9a56784b6b3295490: tablet copy: Beginning tablet copy session from remote peer at address 127.27.17.131:45861
I20260810 22:05:36.063009 3412 tablet_copy_service.cc:140] P bdb9b70cb73a412ab0d4049695126b0a: Received BeginTabletCopySession request for tablet 943ec2cbc29e44e3bcbe0e1e5b039f55 from peer 53b64278bdc549e9a56784b6b3295490 ({username='slave'} at 127.27.17.130:40199)
I20260810 22:05:36.063652 3412 tablet_copy_service.cc:161] P bdb9b70cb73a412ab0d4049695126b0a: Beginning new tablet copy session on tablet 943ec2cbc29e44e3bcbe0e1e5b039f55 from peer 53b64278bdc549e9a56784b6b3295490 at {username='slave'} at 127.27.17.130:40199: session id = 53b64278bdc549e9a56784b6b3295490-943ec2cbc29e44e3bcbe0e1e5b039f55
E20260810 22:05:36.064719 4230 env_posix.cc:432] I/O error, context: INJECTED FAILURE
I20260810 22:05:36.073885 3412 tablet_copy_source_session.cc:205] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:05:36.107822 4320 tablet_copy_client.cc:791] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 53b64278bdc549e9a56784b6b3295490: tablet copy: Starting download of 0 data blocks...
I20260810 22:05:36.108922 4320 tablet_copy_client.cc:655] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 53b64278bdc549e9a56784b6b3295490: tablet copy: Starting download of 1 WAL segments...
I20260810 22:05:36.118752 4320 tablet_copy_client.cc:531] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 53b64278bdc549e9a56784b6b3295490: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:05:36.130529 4320 tablet_bootstrap.cc:492] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 53b64278bdc549e9a56784b6b3295490: Bootstrap starting.
I20260810 22:05:36.172838 4320 tablet_bootstrap.cc:492] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 53b64278bdc549e9a56784b6b3295490: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:05:36.174229 4320 tablet_bootstrap.cc:492] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 53b64278bdc549e9a56784b6b3295490: Bootstrap complete.
I20260810 22:05:36.175833 4320 ts_tablet_manager.cc:1397] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 53b64278bdc549e9a56784b6b3295490: Time spent bootstrapping tablet: real 0.046s user 0.020s sys 0.024s
I20260810 22:05:36.181420 4320 raft_consensus.cc:348] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 53b64278bdc549e9a56784b6b3295490 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: NON_VOTER last_known_addr { host: "127.27.17.130" port: 44557 } attrs { promote: true } }
I20260810 22:05:36.182358 4320 raft_consensus.cc:729] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 53b64278bdc549e9a56784b6b3295490 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 53b64278bdc549e9a56784b6b3295490, State: Initialized, Role: LEARNER
I20260810 22:05:36.183174 4320 consensus_queue.cc:260] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 53b64278bdc549e9a56784b6b3295490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: NON_VOTER last_known_addr { host: "127.27.17.130" port: 44557 } attrs { promote: true } }
I20260810 22:05:36.187189 4320 ts_tablet_manager.cc:1428] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 53b64278bdc549e9a56784b6b3295490: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260810 22:05:36.190517 3412 tablet_copy_service.cc:342] P bdb9b70cb73a412ab0d4049695126b0a: Request end of tablet copy session 53b64278bdc549e9a56784b6b3295490-943ec2cbc29e44e3bcbe0e1e5b039f55 received from {username='slave'} at 127.27.17.130:40199
I20260810 22:05:36.191277 3412 tablet_copy_service.cc:433] P bdb9b70cb73a412ab0d4049695126b0a: ending tablet copy session 53b64278bdc549e9a56784b6b3295490-943ec2cbc29e44e3bcbe0e1e5b039f55 on tablet 943ec2cbc29e44e3bcbe0e1e5b039f55 with peer 53b64278bdc549e9a56784b6b3295490
I20260810 22:05:36.237253 3384 raft_consensus.cc:1203] T 391eba1c39404071969e21ddd5abef69 P bdb9b70cb73a412ab0d4049695126b0a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260810 22:05:36.501255 3099 raft_consensus.cc:1203] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 53b64278bdc549e9a56784b6b3295490 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260810 22:05:36.541514 3690 raft_consensus.cc:1203] T cfb52501fc724335a0c3d3e6a7ee95db P da5bd62264364fd198587af606253dd1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20260810 22:05:36.594126 3067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 22:05:36.599720 4147 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 22:05:36.610692 3658 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 22:05:36.668577 3355 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20260810 22:05:36.714605 4333 raft_consensus.cc:1050] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490: attempting to promote NON_VOTER bdb9b70cb73a412ab0d4049695126b0a to VOTER
I20260810 22:05:36.718521 4333 consensus_queue.cc:237] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } attrs { promote: false } }
I20260810 22:05:36.743690 3690 raft_consensus.cc:1261] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1 [term 2 FOLLOWER]: Refusing update from remote peer 53b64278bdc549e9a56784b6b3295490: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260810 22:05:36.745862 4059 consensus_queue.cc:1035] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 22:05:36.764591 3384 raft_consensus.cc:1261] T 391eba1c39404071969e21ddd5abef69 P bdb9b70cb73a412ab0d4049695126b0a [term 2 LEARNER]: Refusing update from remote peer 53b64278bdc549e9a56784b6b3295490: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260810 22:05:36.771489 3690 raft_consensus.cc:2932] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } attrs { promote: false } } }
I20260810 22:05:36.778282 4059 consensus_queue.cc:1035] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 22:05:36.767843 4333 raft_consensus.cc:2932] T 391eba1c39404071969e21ddd5abef69 P 53b64278bdc549e9a56784b6b3295490 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } attrs { promote: false } } }
I20260810 22:05:36.810796 3385 raft_consensus.cc:2932] T 391eba1c39404071969e21ddd5abef69 P bdb9b70cb73a412ab0d4049695126b0a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } attrs { promote: false } } }
I20260810 22:05:36.839154 2474 catalog_manager.cc:5495] T 391eba1c39404071969e21ddd5abef69 P da5bd62264364fd198587af606253dd1 reported cstate change: config changed from index 3 to 4, bdb9b70cb73a412ab0d4049695126b0a (127.27.17.131) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "53b64278bdc549e9a56784b6b3295490" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } attrs { promote: false } } }
I20260810 22:05:37.000398 4359 raft_consensus.cc:1050] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a: attempting to promote NON_VOTER da5bd62264364fd198587af606253dd1 to VOTER
I20260810 22:05:37.013927 4359 consensus_queue.cc:237] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } attrs { promote: false } }
I20260810 22:05:37.062168 4372 raft_consensus.cc:1050] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a: attempting to promote NON_VOTER 53b64278bdc549e9a56784b6b3295490 to VOTER
I20260810 22:05:37.066558 4372 consensus_queue.cc:237] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } attrs { promote: false } }
I20260810 22:05:37.073099 3690 raft_consensus.cc:1261] T cfb52501fc724335a0c3d3e6a7ee95db P da5bd62264364fd198587af606253dd1 [term 1 LEARNER]: Refusing update from remote peer bdb9b70cb73a412ab0d4049695126b0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 22:05:37.075078 4373 consensus_queue.cc:1035] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 22:05:37.097007 3099 raft_consensus.cc:1261] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490 [term 1 FOLLOWER]: Refusing update from remote peer bdb9b70cb73a412ab0d4049695126b0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260810 22:05:37.099418 4373 consensus_queue.cc:1035] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 22:05:37.109617 4373 raft_consensus.cc:2932] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, da5bd62264364fd198587af606253dd1 (127.27.17.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } attrs { promote: false } } }
I20260810 22:05:37.116961 3099 raft_consensus.cc:2932] T cfb52501fc724335a0c3d3e6a7ee95db P 53b64278bdc549e9a56784b6b3295490 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, da5bd62264364fd198587af606253dd1 (127.27.17.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } attrs { promote: false } } }
I20260810 22:05:37.133713 3690 raft_consensus.cc:2932] T cfb52501fc724335a0c3d3e6a7ee95db P da5bd62264364fd198587af606253dd1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, da5bd62264364fd198587af606253dd1 (127.27.17.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } attrs { promote: false } } }
I20260810 22:05:37.147029 3689 raft_consensus.cc:1261] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 2 FOLLOWER]: Refusing update from remote peer bdb9b70cb73a412ab0d4049695126b0a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260810 22:05:37.147107 3099 raft_consensus.cc:1261] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 53b64278bdc549e9a56784b6b3295490 [term 2 LEARNER]: Refusing update from remote peer bdb9b70cb73a412ab0d4049695126b0a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260810 22:05:37.148646 4359 consensus_queue.cc:1035] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 22:05:37.144944 2473 catalog_manager.cc:5495] T cfb52501fc724335a0c3d3e6a7ee95db P bdb9b70cb73a412ab0d4049695126b0a reported cstate change: config changed from index 3 to 4, da5bd62264364fd198587af606253dd1 (127.27.17.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bdb9b70cb73a412ab0d4049695126b0a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260810 22:05:37.152371 4359 consensus_queue.cc:1035] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 22:05:37.168547 3953 raft_consensus.cc:2932] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P bdb9b70cb73a412ab0d4049695126b0a [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 53b64278bdc549e9a56784b6b3295490 (127.27.17.130) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } attrs { promote: false } } }
I20260810 22:05:37.179255 3690 raft_consensus.cc:2932] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 53b64278bdc549e9a56784b6b3295490 (127.27.17.130) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } attrs { promote: false } } }
I20260810 22:05:37.179440 3098 raft_consensus.cc:2932] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P 53b64278bdc549e9a56784b6b3295490 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 53b64278bdc549e9a56784b6b3295490 (127.27.17.130) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } attrs { promote: false } } }
I20260810 22:05:37.192883 2474 catalog_manager.cc:5495] T 943ec2cbc29e44e3bcbe0e1e5b039f55 P da5bd62264364fd198587af606253dd1 reported cstate change: config changed from index 3 to 4, 53b64278bdc549e9a56784b6b3295490 (127.27.17.130) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "bdb9b70cb73a412ab0d4049695126b0a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bdb9b70cb73a412ab0d4049695126b0a" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 45861 } } peers { permanent_uuid: "da5bd62264364fd198587af606253dd1" member_type: VOTER last_known_addr { host: "127.27.17.132" port: 41969 } } peers { permanent_uuid: "53b64278bdc549e9a56784b6b3295490" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 44557 } attrs { promote: false } } }
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
12ce8059216a4cb68b7ba53cac315b54 | 127.27.17.190:33447 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:37169 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
53b64278bdc549e9a56784b6b3295490 | 127.27.17.130:44557 | HEALTHY | <none> | 1 | 0
742853e1f3ae4ac1b86265b6adb193f9 | 127.27.17.129:33615 | HEALTHY | <none> | 0 | 0
bdb9b70cb73a412ab0d4049695126b0a | 127.27.17.131:45861 | HEALTHY | <none> | 3 | 0
da5bd62264364fd198587af606253dd1 | 127.27.17.132:41969 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.27.17.129:33615
env_inject_eio | 1 | hidden | 127.27.17.129:33615
env_inject_eio_globs | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-1/** | hidden | 127.27.17.129:33615
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.27.17.129 | experimental | 127.27.17.129:33615
local_ip_for_outbound_sockets | 127.27.17.130 | experimental | 127.27.17.130:44557
local_ip_for_outbound_sockets | 127.27.17.131 | experimental | 127.27.17.131:45861
local_ip_for_outbound_sockets | 127.27.17.132 | experimental | 127.27.17.132:41969
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.27.17.129:33615
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.27.17.130:44557
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.27.17.131:45861
server_dump_info_path | /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786399438989427-27718-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.27.17.132:41969
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.27.17.148:37169 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 5 | 5 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 742853e1f3ae4ac1b86265b6adb193f9 | 127.27.17.129:33615 | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260810 22:05:37.216766 27718 log_verifier.cc:126] Checking tablet 159f82c64cc8402fb2570de1b5223e78
I20260810 22:05:37.281306 27718 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 159f82c64cc8402fb2570de1b5223e78
I20260810 22:05:37.281540 27718 log_verifier.cc:126] Checking tablet 391eba1c39404071969e21ddd5abef69
I20260810 22:05:37.346832 27718 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 391eba1c39404071969e21ddd5abef69
I20260810 22:05:37.347045 27718 log_verifier.cc:126] Checking tablet 7230e7ad4d70476e85b798e58579d61f
I20260810 22:05:37.406301 27718 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 7230e7ad4d70476e85b798e58579d61f
I20260810 22:05:37.406520 27718 log_verifier.cc:126] Checking tablet 943ec2cbc29e44e3bcbe0e1e5b039f55
I20260810 22:05:37.472534 27718 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 943ec2cbc29e44e3bcbe0e1e5b039f55
I20260810 22:05:37.472818 27718 log_verifier.cc:126] Checking tablet cfb52501fc724335a0c3d3e6a7ee95db
I20260810 22:05:37.540138 27718 log_verifier.cc:177] Verified matching terms for 4 ops in tablet cfb52501fc724335a0c3d3e6a7ee95db
I20260810 22:05:38.082986 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 3910
I20260810 22:05:38.172555 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 2863
W20260810 22:05:38.224769 3294 connection.cc:535] client connection to 127.27.17.130:44557 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 22:05:38.225260 3294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 15 similar messages]
W20260810 22:05:38.227325 3599 connection.cc:535] client connection to 127.27.17.130:44557 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 22:05:38.227782 3599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 22:05:38.228978 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 3164
W20260810 22:05:38.243628 3599 consensus_peers.cc:487] T 7230e7ad4d70476e85b798e58579d61f P da5bd62264364fd198587af606253dd1 -> Peer 53b64278bdc549e9a56784b6b3295490 (127.27.17.130:44557): Couldn't send request to peer 53b64278bdc549e9a56784b6b3295490. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.130:44557: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 22:05:38.281296 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 3455
I20260810 22:05:38.335017 27718 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu with pid 2308
2026-08-10T22:05:38Z chronyd exiting
[ OK ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/5 (19680 ms)
[----------] 6 tests from DiskFailure/DiskFailureITest (99158 ms total)
[----------] 6 tests from TabletServerDiskError/TabletServerDiskErrorITest
[ RUN ] TabletServerDiskError/TabletServerDiskErrorITest.TestFailOnBootstrap/0
2026-08-10T22:05:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:05:38Z Disabled control of system clock
I20260810 22:05:38.482348 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/data-2
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.190:44453
--webserver_interface=127.27.17.190
--webserver_port=0
--builtin_ntp_servers=127.27.17.148:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.190:44453
--master_tombstone_evicted_tablet_replicas=false with env {}
W20260810 22:05:38.798133 4413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:05:38.798761 4413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:05:38.799186 4413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:05:38.830819 4413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:05:38.831130 4413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:05:38.831344 4413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:05:38.831544 4413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:05:38.867503 4413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/data-2
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/wal
--master_tombstone_evicted_tablet_replicas=false
--ipki_ca_key_size=768
--master_addresses=127.27.17.190:44453
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.190:44453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
--webserver_interface=127.27.17.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:05:38.868686 4413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:05:38.870450 4413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:05:38.890620 4427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:38.890635 4421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:38.891930 4413 server_base.cc:1034] running on GCE node
W20260810 22:05:38.890714 4422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:40.094681 4413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:05:40.097530 4413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:05:40.098896 4413 hybrid_clock.cc:648] HybridClock initialized: now 1786399540098845 us; error 74 us; skew 500 ppm
I20260810 22:05:40.099757 4413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:05:40.106801 4413 webserver.cc:460] Webserver started at http://127.27.17.190:34317/ using document root <none> and password file <none>
I20260810 22:05:40.108003 4413 fs_manager.cc:359] Metadata directory not provided
I20260810 22:05:40.108196 4413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:05:40.108701 4413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:05:40.118095 4413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/data-0/instance:
uuid: "ab364b187b554eaca851658e3f407e72"
format_stamp: "Formatted at 2026-08-10 22:05:40 on dist-test-slave-bw88"
I20260810 22:05:40.119220 4413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/data-1/instance:
uuid: "ab364b187b554eaca851658e3f407e72"
format_stamp: "Formatted at 2026-08-10 22:05:40 on dist-test-slave-bw88"
I20260810 22:05:40.120080 4413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/data-2/instance:
uuid: "ab364b187b554eaca851658e3f407e72"
format_stamp: "Formatted at 2026-08-10 22:05:40 on dist-test-slave-bw88"
I20260810 22:05:40.120910 4413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/wal/instance:
uuid: "ab364b187b554eaca851658e3f407e72"
format_stamp: "Formatted at 2026-08-10 22:05:40 on dist-test-slave-bw88"
I20260810 22:05:40.137100 4413 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.010s sys 0.005s
I20260810 22:05:40.144470 4441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:40.144976 4441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:40.146184 4441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:40.147485 4413 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.002s
I20260810 22:05:40.147853 4413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/wal
uuid: "ab364b187b554eaca851658e3f407e72"
format_stamp: "Formatted at 2026-08-10 22:05:40 on dist-test-slave-bw88"
I20260810 22:05:40.148231 4413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/wal
3 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/data-2/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:05:40.180838 4413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:05:40.182468 4413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:05:40.182960 4413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:05:40.258821 4413 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.190:44453
I20260810 22:05:40.258934 4517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.190:44453 every 8 connection(s)
I20260810 22:05:40.261679 4413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/data-0/info.pb
I20260810 22:05:40.267539 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 4413
I20260810 22:05:40.268214 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/master-0/wal/instance
I20260810 22:05:40.294207 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72: Bootstrap starting.
I20260810 22:05:40.299731 4518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:40.301999 4518 log.cc:824] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72: Log is configured to *not* fsync() on all Append() calls
I20260810 22:05:40.307240 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72: No bootstrap required, opened a new log
I20260810 22:05:40.327486 4518 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab364b187b554eaca851658e3f407e72" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 44453 } }
I20260810 22:05:40.328190 4518 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:40.328497 4518 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab364b187b554eaca851658e3f407e72, State: Initialized, Role: FOLLOWER
I20260810 22:05:40.329463 4518 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab364b187b554eaca851658e3f407e72" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 44453 } }
I20260810 22:05:40.330046 4518 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:05:40.330361 4518 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:05:40.330662 4518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:40.334647 4518 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab364b187b554eaca851658e3f407e72" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 44453 } }
I20260810 22:05:40.335374 4518 leader_election.cc:302] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab364b187b554eaca851658e3f407e72; no voters:
I20260810 22:05:40.337608 4518 leader_election.cc:288] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:05:40.338009 4523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:40.341687 4523 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72 [term 1 LEADER]: Becoming Leader. State: Replica: ab364b187b554eaca851658e3f407e72, State: Running, Role: LEADER
I20260810 22:05:40.342612 4523 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab364b187b554eaca851658e3f407e72" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 44453 } }
I20260810 22:05:40.344274 4518 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:05:40.355275 4524 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab364b187b554eaca851658e3f407e72. Latest consensus state: current_term: 1 leader_uuid: "ab364b187b554eaca851658e3f407e72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab364b187b554eaca851658e3f407e72" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 44453 } } }
I20260810 22:05:40.355547 4526 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab364b187b554eaca851658e3f407e72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab364b187b554eaca851658e3f407e72" member_type: VOTER last_known_addr { host: "127.27.17.190" port: 44453 } } }
I20260810 22:05:40.356735 4524 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72 [sys.catalog]: This master's current role is: LEADER
I20260810 22:05:40.356802 4526 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72 [sys.catalog]: This master's current role is: LEADER
I20260810 22:05:40.359735 4536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:05:40.372714 4536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:05:40.391711 4536 catalog_manager.cc:1347] Generated new cluster ID: 638a394010fe4c23a35644ad034e264c
I20260810 22:05:40.392012 4536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:05:40.426695 4536 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:05:40.428493 4536 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:05:40.444000 4536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab364b187b554eaca851658e3f407e72: Generated new TSK 0
I20260810 22:05:40.445411 4536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:05:40.467578 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-2
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.129:0
--local_ip_for_outbound_sockets=127.27.17.129
--webserver_interface=127.27.17.129
--webserver_port=0
--tserver_master_addrs=127.27.17.190:44453
--builtin_ntp_servers=127.27.17.148:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_secs=1
--flush_threshold_mb=1 with env {}
W20260810 22:05:40.924796 4551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:05:40.925557 4551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:05:40.926154 4551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:05:40.959820 4551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:05:40.960431 4551 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260810 22:05:40.960667 4551 flags.cc:425] Enabled experimental flag: --flush_threshold_secs=1
W20260810 22:05:40.961215 4551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.129
I20260810 22:05:40.998008 4551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-2
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.27.17.129
--webserver_port=0
--flush_threshold_mb=1
--flush_threshold_secs=1
--tserver_master_addrs=127.27.17.190:44453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.129
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:05:40.999293 4551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:05:41.000941 4551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:05:41.045367 4565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:41.057921 4568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:41.048419 4566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:42.355677 4567 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1312 milliseconds
I20260810 22:05:42.355862 4551 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 22:05:42.357554 4551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:05:42.360472 4551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:05:42.361937 4551 hybrid_clock.cc:648] HybridClock initialized: now 1786399542361901 us; error 41 us; skew 500 ppm
I20260810 22:05:42.363024 4551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:05:42.374366 4551 webserver.cc:460] Webserver started at http://127.27.17.129:33345/ using document root <none> and password file <none>
I20260810 22:05:42.376142 4551 fs_manager.cc:359] Metadata directory not provided
I20260810 22:05:42.376441 4551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:05:42.377208 4551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:05:42.384529 4551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/instance:
uuid: "2b27c8ff42ea450bb5114ab97c1ed25d"
format_stamp: "Formatted at 2026-08-10 22:05:42 on dist-test-slave-bw88"
I20260810 22:05:42.386089 4551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1/instance:
uuid: "2b27c8ff42ea450bb5114ab97c1ed25d"
format_stamp: "Formatted at 2026-08-10 22:05:42 on dist-test-slave-bw88"
I20260810 22:05:42.387305 4551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-2/instance:
uuid: "2b27c8ff42ea450bb5114ab97c1ed25d"
format_stamp: "Formatted at 2026-08-10 22:05:42 on dist-test-slave-bw88"
I20260810 22:05:42.388511 4551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/wal/instance:
uuid: "2b27c8ff42ea450bb5114ab97c1ed25d"
format_stamp: "Formatted at 2026-08-10 22:05:42 on dist-test-slave-bw88"
I20260810 22:05:42.404523 4551 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.009s sys 0.008s
I20260810 22:05:42.411633 4584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:42.412210 4584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:42.413206 4584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:42.414568 4551 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.007s sys 0.001s
I20260810 22:05:42.414904 4551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/wal
uuid: "2b27c8ff42ea450bb5114ab97c1ed25d"
format_stamp: "Formatted at 2026-08-10 22:05:42 on dist-test-slave-bw88"
I20260810 22:05:42.415298 4551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/wal
3 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-2/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:05:42.452373 4551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:05:42.454017 4551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:05:42.454514 4551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:05:42.457731 4551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:05:42.462637 4551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:05:42.462888 4551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:42.463160 4551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:05:42.463341 4551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:42.643158 4551 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.129:43093
I20260810 22:05:42.643328 4749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.129:43093 every 8 connection(s)
I20260810 22:05:42.645980 4551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/info.pb
I20260810 22:05:42.653235 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 4551
I20260810 22:05:42.653861 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/wal/instance
I20260810 22:05:42.662493 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-2
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.130:0
--local_ip_for_outbound_sockets=127.27.17.130
--webserver_interface=127.27.17.130
--webserver_port=0
--tserver_master_addrs=127.27.17.190:44453
--builtin_ntp_servers=127.27.17.148:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_secs=1
--flush_threshold_mb=1 with env {}
I20260810 22:05:42.678658 4751 heartbeater.cc:347] Connected to a master server at 127.27.17.190:44453
I20260810 22:05:42.679098 4751 heartbeater.cc:464] Registering TS with master...
I20260810 22:05:42.680061 4751 heartbeater.cc:511] Master 127.27.17.190:44453 requested a full tablet report, sending...
I20260810 22:05:42.682772 4472 ts_manager.cc:194] Registered new tserver with Master: 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093)
I20260810 22:05:42.685082 4472 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.129:36529
W20260810 22:05:42.996958 4756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:05:42.997545 4756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:05:42.998063 4756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:05:43.028342 4756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:05:43.028883 4756 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260810 22:05:43.029062 4756 flags.cc:425] Enabled experimental flag: --flush_threshold_secs=1
W20260810 22:05:43.029524 4756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.130
I20260810 22:05:43.064862 4756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-2
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
--webserver_interface=127.27.17.130
--webserver_port=0
--flush_threshold_mb=1
--flush_threshold_secs=1
--tserver_master_addrs=127.27.17.190:44453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.130
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:05:43.066133 4756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:05:43.067744 4756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:05:43.086045 4762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:43.689002 4751 heartbeater.cc:503] Master 127.27.17.190:44453 was elected leader, sending a full tablet report...
W20260810 22:05:43.089612 4763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:43.086190 4765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:44.302470 4764 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1214 milliseconds
I20260810 22:05:44.302588 4756 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 22:05:44.303881 4756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:05:44.306689 4756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:05:44.308125 4756 hybrid_clock.cc:648] HybridClock initialized: now 1786399544308070 us; error 77 us; skew 500 ppm
I20260810 22:05:44.308881 4756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:05:44.315848 4756 webserver.cc:460] Webserver started at http://127.27.17.130:37233/ using document root <none> and password file <none>
I20260810 22:05:44.317035 4756 fs_manager.cc:359] Metadata directory not provided
I20260810 22:05:44.317229 4756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:05:44.317797 4756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:05:44.322643 4756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0/instance:
uuid: "b9235c169d85402faf437d72e9a38bdd"
format_stamp: "Formatted at 2026-08-10 22:05:44 on dist-test-slave-bw88"
I20260810 22:05:44.323685 4756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-1/instance:
uuid: "b9235c169d85402faf437d72e9a38bdd"
format_stamp: "Formatted at 2026-08-10 22:05:44 on dist-test-slave-bw88"
I20260810 22:05:44.324514 4756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-2/instance:
uuid: "b9235c169d85402faf437d72e9a38bdd"
format_stamp: "Formatted at 2026-08-10 22:05:44 on dist-test-slave-bw88"
I20260810 22:05:44.325394 4756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/wal/instance:
uuid: "b9235c169d85402faf437d72e9a38bdd"
format_stamp: "Formatted at 2026-08-10 22:05:44 on dist-test-slave-bw88"
I20260810 22:05:44.340547 4756 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.010s sys 0.004s
I20260810 22:05:44.347352 4777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:44.347844 4777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:44.349056 4778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:44.350709 4756 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.005s
I20260810 22:05:44.351090 4756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/wal
uuid: "b9235c169d85402faf437d72e9a38bdd"
format_stamp: "Formatted at 2026-08-10 22:05:44 on dist-test-slave-bw88"
I20260810 22:05:44.351516 4756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/wal
3 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-2/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:05:44.398313 4756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:05:44.399837 4756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:05:44.400321 4756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:05:44.402918 4756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:05:44.407186 4756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:05:44.407449 4756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:44.407713 4756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:05:44.407896 4756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:44.552752 4756 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.130:37933
I20260810 22:05:44.552852 4934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.130:37933 every 8 connection(s)
I20260810 22:05:44.555581 4756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0/info.pb
I20260810 22:05:44.565521 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 4756
I20260810 22:05:44.566130 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/wal/instance
I20260810 22:05:44.574342 27718 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
/tmp/dist-test-taskld5GkP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-2
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.131:0
--local_ip_for_outbound_sockets=127.27.17.131
--webserver_interface=127.27.17.131
--webserver_port=0
--tserver_master_addrs=127.27.17.190:44453
--builtin_ntp_servers=127.27.17.148:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_secs=1
--flush_threshold_mb=1 with env {}
I20260810 22:05:44.584406 4939 heartbeater.cc:347] Connected to a master server at 127.27.17.190:44453
I20260810 22:05:44.584867 4939 heartbeater.cc:464] Registering TS with master...
I20260810 22:05:44.585911 4939 heartbeater.cc:511] Master 127.27.17.190:44453 requested a full tablet report, sending...
I20260810 22:05:44.588299 4472 ts_manager.cc:194] Registered new tserver with Master: b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
I20260810 22:05:44.589841 4472 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.130:35445
W20260810 22:05:44.895988 4944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:05:44.896574 4944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:05:44.897176 4944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:05:44.928043 4944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:05:44.928682 4944 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260810 22:05:44.928933 4944 flags.cc:425] Enabled experimental flag: --flush_threshold_secs=1
W20260810 22:05:44.929476 4944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.131
I20260810 22:05:44.965015 4944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.148:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-2
--fs_wal_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
--webserver_interface=127.27.17.131
--webserver_port=0
--flush_threshold_mb=1
--flush_threshold_secs=1
--tserver_master_addrs=127.27.17.190:44453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.131
--log_dir=/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 21:15:18 UTC on ccycloud.quasar-rkqfvv.root.comops.site
build id 4579042
TSAN enabled
I20260810 22:05:44.966363 4944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:05:44.968093 4944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:05:44.987444 4951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:05:45.593421 4939 heartbeater.cc:503] Master 127.27.17.190:44453 was elected leader, sending a full tablet report...
W20260810 22:05:44.987919 4953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:44.987439 4950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:05:46.253444 4952 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260810 22:05:46.253626 4944 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260810 22:05:46.257069 4944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:05:46.260213 4944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:05:46.261633 4944 hybrid_clock.cc:648] HybridClock initialized: now 1786399546261576 us; error 79 us; skew 500 ppm
I20260810 22:05:46.262692 4944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 22:05:46.271525 4944 webserver.cc:460] Webserver started at http://127.27.17.131:36991/ using document root <none> and password file <none>
I20260810 22:05:46.272838 4944 fs_manager.cc:359] Metadata directory not provided
I20260810 22:05:46.273092 4944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:05:46.273725 4944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:05:46.278774 4944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0/instance:
uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc"
format_stamp: "Formatted at 2026-08-10 22:05:46 on dist-test-slave-bw88"
I20260810 22:05:46.279896 4944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-1/instance:
uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc"
format_stamp: "Formatted at 2026-08-10 22:05:46 on dist-test-slave-bw88"
I20260810 22:05:46.280805 4944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-2/instance:
uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc"
format_stamp: "Formatted at 2026-08-10 22:05:46 on dist-test-slave-bw88"
I20260810 22:05:46.281790 4944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/wal/instance:
uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc"
format_stamp: "Formatted at 2026-08-10 22:05:46 on dist-test-slave-bw88"
I20260810 22:05:46.297524 4944 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.012s sys 0.004s
I20260810 22:05:46.304270 4965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:46.304812 4965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:46.306032 4966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:46.307394 4944 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260810 22:05:46.307679 4944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/wal
uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc"
format_stamp: "Formatted at 2026-08-10 22:05:46 on dist-test-slave-bw88"
I20260810 22:05:46.308013 4944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/wal
3 data directories: /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-2/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:05:46.349782 4944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:05:46.351235 4944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:05:46.351666 4944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:05:46.354254 4944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:05:46.358693 4944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:05:46.358948 4944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:46.359216 4944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:05:46.359393 4944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:05:46.510357 4944 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.131:44793
I20260810 22:05:46.510445 5122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.17.131:44793 every 8 connection(s)
I20260810 22:05:46.513473 4944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0/info.pb
I20260810 22:05:46.521526 27718 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskld5GkP/build/tsan/bin/kudu as pid 4944
I20260810 22:05:46.522038 27718 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/wal/instance
I20260810 22:05:46.538725 5123 heartbeater.cc:347] Connected to a master server at 127.27.17.190:44453
I20260810 22:05:46.539179 5123 heartbeater.cc:464] Registering TS with master...
I20260810 22:05:46.540241 5123 heartbeater.cc:511] Master 127.27.17.190:44453 requested a full tablet report, sending...
I20260810 22:05:46.542755 4472 ts_manager.cc:194] Registered new tserver with Master: a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:46.544224 4472 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.17.131:37527
I20260810 22:05:46.557802 27718 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:05:46.593243 27718 test_util.cc:268] Using random seed: -1905864765
I20260810 22:05:46.637511 4472 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46786:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 10
seed: 0
}
}
W20260810 22:05:46.641893 4472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:05:46.739465 4839 tablet_service.cc:1467] Processing CreateTablet for tablet e33e69aaf4a64216b2f995a9e9e9e1e2 (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.742333 4836 tablet_service.cc:1467] Processing CreateTablet for tablet af1fb81326b54dbbb23c7b577bdaf751 (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.745792 4834 tablet_service.cc:1467] Processing CreateTablet for tablet 518b19decff74b1baf253900da181d46 (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.747493 4835 tablet_service.cc:1467] Processing CreateTablet for tablet 778d8c794af64277b2f497708012e16c (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.748598 4840 tablet_service.cc:1467] Processing CreateTablet for tablet dc4801ac5aeb4b4d8340ee2f5c0d73f6 (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.749158 4838 tablet_service.cc:1467] Processing CreateTablet for tablet 628379a2077c4ea7896784b089a5ea0c (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.757211 4654 tablet_service.cc:1467] Processing CreateTablet for tablet 628379a2077c4ea7896784b089a5ea0c (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.756652 4656 tablet_service.cc:1467] Processing CreateTablet for tablet dc4801ac5aeb4b4d8340ee2f5c0d73f6 (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.757228 4655 tablet_service.cc:1467] Processing CreateTablet for tablet e33e69aaf4a64216b2f995a9e9e9e1e2 (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.758693 4832 tablet_service.cc:1467] Processing CreateTablet for tablet 8e619eaa340248da8a54a913c1d16b9f (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.759145 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 778d8c794af64277b2f497708012e16c (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.757450 4653 tablet_service.cc:1467] Processing CreateTablet for tablet af1fb81326b54dbbb23c7b577bdaf751 (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.760402 4830 tablet_service.cc:1467] Processing CreateTablet for tablet df71b8fe94f84062bbf858f1cc4a0920 (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.761740 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 518b19decff74b1baf253900da181d46 (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.763249 4829 tablet_service.cc:1467] Processing CreateTablet for tablet f0037195580e420fad2a77957025fc74 (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.763585 4828 tablet_service.cc:1467] Processing CreateTablet for tablet 416ba6bb2af24654a64eb1a23261254b (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.769049 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 8e619eaa340248da8a54a913c1d16b9f (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.773795 4647 tablet_service.cc:1467] Processing CreateTablet for tablet f0037195580e420fad2a77957025fc74 (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.774716 5027 tablet_service.cc:1467] Processing CreateTablet for tablet dc4801ac5aeb4b4d8340ee2f5c0d73f6 (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.778661 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 416ba6bb2af24654a64eb1a23261254b (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.776321 4648 tablet_service.cc:1467] Processing CreateTablet for tablet df71b8fe94f84062bbf858f1cc4a0920 (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.779603 5025 tablet_service.cc:1467] Processing CreateTablet for tablet e33e69aaf4a64216b2f995a9e9e9e1e2 (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.787287 5023 tablet_service.cc:1467] Processing CreateTablet for tablet 628379a2077c4ea7896784b089a5ea0c (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.788805 5019 tablet_service.cc:1467] Processing CreateTablet for tablet 8e619eaa340248da8a54a913c1d16b9f (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.790750 5022 tablet_service.cc:1467] Processing CreateTablet for tablet af1fb81326b54dbbb23c7b577bdaf751 (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.798084 5021 tablet_service.cc:1467] Processing CreateTablet for tablet 778d8c794af64277b2f497708012e16c (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.800626 5018 tablet_service.cc:1467] Processing CreateTablet for tablet df71b8fe94f84062bbf858f1cc4a0920 (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.806473 5020 tablet_service.cc:1467] Processing CreateTablet for tablet 518b19decff74b1baf253900da181d46 (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.809413 5016 tablet_service.cc:1467] Processing CreateTablet for tablet 416ba6bb2af24654a64eb1a23261254b (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.810760 5017 tablet_service.cc:1467] Processing CreateTablet for tablet f0037195580e420fad2a77957025fc74 (DEFAULT_TABLE table=test-workload [id=c35860d37ed640daab2f5e62ffc9c9e8]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260810 22:05:46.882288 5155 tablet_bootstrap.cc:492] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d: Bootstrap starting.
I20260810 22:05:46.890261 5155 tablet_bootstrap.cc:654] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:46.892788 5155 log.cc:824] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d: Log is configured to *not* fsync() on all Append() calls
I20260810 22:05:46.940846 5156 tablet_bootstrap.cc:492] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd: Bootstrap starting.
I20260810 22:05:46.942742 5154 tablet_bootstrap.cc:492] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d: Bootstrap starting.
I20260810 22:05:46.947908 5153 tablet_bootstrap.cc:492] T 628379a2077c4ea7896784b089a5ea0c P 2b27c8ff42ea450bb5114ab97c1ed25d: Bootstrap starting.
I20260810 22:05:46.966576 5154 tablet_bootstrap.cc:654] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:46.967141 5161 tablet_bootstrap.cc:492] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd: Bootstrap starting.
I20260810 22:05:46.968065 5159 tablet_bootstrap.cc:492] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd: Bootstrap starting.
I20260810 22:05:46.970132 5153 tablet_bootstrap.cc:654] T 628379a2077c4ea7896784b089a5ea0c P 2b27c8ff42ea450bb5114ab97c1ed25d: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:46.985217 5154 tablet_bootstrap.cc:492] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d: No bootstrap required, opened a new log
I20260810 22:05:46.985244 5153 tablet_bootstrap.cc:492] T 628379a2077c4ea7896784b089a5ea0c P 2b27c8ff42ea450bb5114ab97c1ed25d: No bootstrap required, opened a new log
I20260810 22:05:46.986018 5153 ts_tablet_manager.cc:1397] T 628379a2077c4ea7896784b089a5ea0c P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.010s
I20260810 22:05:46.986055 5154 ts_tablet_manager.cc:1397] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.005s
I20260810 22:05:46.992030 5166 tablet_bootstrap.cc:492] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Bootstrap starting.
I20260810 22:05:46.996270 5156 tablet_bootstrap.cc:654] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:46.996459 5155 tablet_bootstrap.cc:492] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d: No bootstrap required, opened a new log
I20260810 22:05:46.996878 5155 ts_tablet_manager.cc:1397] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent bootstrapping tablet: real 0.115s user 0.045s sys 0.025s
I20260810 22:05:46.998782 5156 log.cc:824] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd: Log is configured to *not* fsync() on all Append() calls
I20260810 22:05:46.999570 5159 tablet_bootstrap.cc:654] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.000022 5161 tablet_bootstrap.cc:654] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.010625 5159 tablet_bootstrap.cc:492] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd: No bootstrap required, opened a new log
I20260810 22:05:47.011193 5159 ts_tablet_manager.cc:1397] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.011s
I20260810 22:05:47.013015 5163 tablet_bootstrap.cc:492] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Bootstrap starting.
I20260810 22:05:47.015884 5161 tablet_bootstrap.cc:492] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd: No bootstrap required, opened a new log
I20260810 22:05:47.016314 5161 ts_tablet_manager.cc:1397] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.005s
I20260810 22:05:47.022296 5156 tablet_bootstrap.cc:492] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd: No bootstrap required, opened a new log
I20260810 22:05:47.022696 5156 ts_tablet_manager.cc:1397] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd: Time spent bootstrapping tablet: real 0.082s user 0.004s sys 0.011s
I20260810 22:05:47.023563 5154 raft_consensus.cc:348] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.024633 5154 raft_consensus.cc:374] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.025043 5154 raft_consensus.cc:729] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b27c8ff42ea450bb5114ab97c1ed25d, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.026284 5154 consensus_queue.cc:260] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.029471 5167 tablet_bootstrap.cc:492] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Bootstrap starting.
I20260810 22:05:47.027688 5155 raft_consensus.cc:348] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.029935 5155 raft_consensus.cc:374] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.030359 5155 raft_consensus.cc:729] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b27c8ff42ea450bb5114ab97c1ed25d, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.031251 5155 consensus_queue.cc:260] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.034893 5154 ts_tablet_manager.cc:1428] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent starting tablet: real 0.048s user 0.024s sys 0.009s
I20260810 22:05:47.035851 5154 tablet_bootstrap.cc:492] T 518b19decff74b1baf253900da181d46 P 2b27c8ff42ea450bb5114ab97c1ed25d: Bootstrap starting.
I20260810 22:05:47.041469 5153 raft_consensus.cc:348] T 628379a2077c4ea7896784b089a5ea0c P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.042280 5153 raft_consensus.cc:374] T 628379a2077c4ea7896784b089a5ea0c P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.042559 5153 raft_consensus.cc:729] T 628379a2077c4ea7896784b089a5ea0c P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b27c8ff42ea450bb5114ab97c1ed25d, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.043797 5154 tablet_bootstrap.cc:654] T 518b19decff74b1baf253900da181d46 P 2b27c8ff42ea450bb5114ab97c1ed25d: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.044575 5163 tablet_bootstrap.cc:654] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.047062 5163 log.cc:824] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Log is configured to *not* fsync() on all Append() calls
I20260810 22:05:47.049160 5153 consensus_queue.cc:260] T 628379a2077c4ea7896784b089a5ea0c P 2b27c8ff42ea450bb5114ab97c1ed25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.052338 5167 tablet_bootstrap.cc:654] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.052171 5156 raft_consensus.cc:348] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.053186 5156 raft_consensus.cc:374] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.053553 5156 raft_consensus.cc:729] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9235c169d85402faf437d72e9a38bdd, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.054517 5156 consensus_queue.cc:260] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.057578 5153 ts_tablet_manager.cc:1428] T 628379a2077c4ea7896784b089a5ea0c P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent starting tablet: real 0.071s user 0.002s sys 0.006s
I20260810 22:05:47.041244 5166 tablet_bootstrap.cc:654] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.057720 5154 tablet_bootstrap.cc:492] T 518b19decff74b1baf253900da181d46 P 2b27c8ff42ea450bb5114ab97c1ed25d: No bootstrap required, opened a new log
I20260810 22:05:47.058522 5154 ts_tablet_manager.cc:1397] T 518b19decff74b1baf253900da181d46 P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.013s
I20260810 22:05:47.061936 5154 raft_consensus.cc:348] T 518b19decff74b1baf253900da181d46 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.058982 5161 raft_consensus.cc:348] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.063627 5163 tablet_bootstrap.cc:492] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc: No bootstrap required, opened a new log
I20260810 22:05:47.063750 5161 raft_consensus.cc:374] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.063640 5154 raft_consensus.cc:374] T 518b19decff74b1baf253900da181d46 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.064019 5161 raft_consensus.cc:729] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9235c169d85402faf437d72e9a38bdd, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.064153 5163 ts_tablet_manager.cc:1397] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent bootstrapping tablet: real 0.099s user 0.011s sys 0.003s
I20260810 22:05:47.064235 5154 raft_consensus.cc:729] T 518b19decff74b1baf253900da181d46 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b27c8ff42ea450bb5114ab97c1ed25d, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.064707 5161 consensus_queue.cc:260] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.065363 5154 consensus_queue.cc:260] T 518b19decff74b1baf253900da181d46 P 2b27c8ff42ea450bb5114ab97c1ed25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.068372 5153 tablet_bootstrap.cc:492] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d: Bootstrap starting.
I20260810 22:05:47.069748 5154 ts_tablet_manager.cc:1428] T 518b19decff74b1baf253900da181d46 P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent starting tablet: real 0.011s user 0.007s sys 0.001s
I20260810 22:05:47.067782 5159 raft_consensus.cc:348] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } } peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } }
I20260810 22:05:47.070284 5159 raft_consensus.cc:374] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.070580 5159 raft_consensus.cc:729] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9235c169d85402faf437d72e9a38bdd, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.072400 5154 tablet_bootstrap.cc:492] T df71b8fe94f84062bbf858f1cc4a0920 P 2b27c8ff42ea450bb5114ab97c1ed25d: Bootstrap starting.
I20260810 22:05:47.073823 5159 consensus_queue.cc:260] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } } peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } }
I20260810 22:05:47.076859 5155 ts_tablet_manager.cc:1428] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent starting tablet: real 0.080s user 0.011s sys 0.035s
I20260810 22:05:47.077766 5155 tablet_bootstrap.cc:492] T 778d8c794af64277b2f497708012e16c P 2b27c8ff42ea450bb5114ab97c1ed25d: Bootstrap starting.
I20260810 22:05:47.079562 5161 ts_tablet_manager.cc:1428] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd: Time spent starting tablet: real 0.063s user 0.010s sys 0.007s
I20260810 22:05:47.080571 5161 tablet_bootstrap.cc:492] T 416ba6bb2af24654a64eb1a23261254b P b9235c169d85402faf437d72e9a38bdd: Bootstrap starting.
I20260810 22:05:47.083334 5154 tablet_bootstrap.cc:654] T df71b8fe94f84062bbf858f1cc4a0920 P 2b27c8ff42ea450bb5114ab97c1ed25d: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.083662 5155 tablet_bootstrap.cc:654] T 778d8c794af64277b2f497708012e16c P 2b27c8ff42ea450bb5114ab97c1ed25d: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.083832 5156 ts_tablet_manager.cc:1428] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd: Time spent starting tablet: real 0.061s user 0.010s sys 0.003s
I20260810 22:05:47.084736 5156 tablet_bootstrap.cc:492] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd: Bootstrap starting.
I20260810 22:05:47.087888 5159 ts_tablet_manager.cc:1428] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd: Time spent starting tablet: real 0.076s user 0.026s sys 0.005s
I20260810 22:05:47.088788 5159 tablet_bootstrap.cc:492] T af1fb81326b54dbbb23c7b577bdaf751 P b9235c169d85402faf437d72e9a38bdd: Bootstrap starting.
I20260810 22:05:47.097808 5167 tablet_bootstrap.cc:492] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc: No bootstrap required, opened a new log
I20260810 22:05:47.098294 5167 ts_tablet_manager.cc:1397] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.009s
I20260810 22:05:47.099737 5156 tablet_bootstrap.cc:654] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.104187 5153 tablet_bootstrap.cc:654] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.107846 5166 tablet_bootstrap.cc:492] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc: No bootstrap required, opened a new log
I20260810 22:05:47.108278 5166 ts_tablet_manager.cc:1397] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent bootstrapping tablet: real 0.117s user 0.006s sys 0.006s
I20260810 22:05:47.108599 5161 tablet_bootstrap.cc:654] T 416ba6bb2af24654a64eb1a23261254b P b9235c169d85402faf437d72e9a38bdd: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.111866 5188 raft_consensus.cc:482] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:47.112435 5188 raft_consensus.cc:504] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.118831 5154 tablet_bootstrap.cc:492] T df71b8fe94f84062bbf858f1cc4a0920 P 2b27c8ff42ea450bb5114ab97c1ed25d: No bootstrap required, opened a new log
I20260810 22:05:47.119005 5155 tablet_bootstrap.cc:492] T 778d8c794af64277b2f497708012e16c P 2b27c8ff42ea450bb5114ab97c1ed25d: No bootstrap required, opened a new log
I20260810 22:05:47.119285 5154 ts_tablet_manager.cc:1397] T df71b8fe94f84062bbf858f1cc4a0920 P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent bootstrapping tablet: real 0.047s user 0.014s sys 0.009s
I20260810 22:05:47.119460 5155 ts_tablet_manager.cc:1397] T 778d8c794af64277b2f497708012e16c P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.012s
I20260810 22:05:47.120705 5188 leader_election.cc:288] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:47.122102 5155 raft_consensus.cc:348] T 778d8c794af64277b2f497708012e16c P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.122821 5155 raft_consensus.cc:374] T 778d8c794af64277b2f497708012e16c P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.123116 5155 raft_consensus.cc:729] T 778d8c794af64277b2f497708012e16c P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b27c8ff42ea450bb5114ab97c1ed25d, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.123767 5155 consensus_queue.cc:260] T 778d8c794af64277b2f497708012e16c P 2b27c8ff42ea450bb5114ab97c1ed25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.124756 5154 raft_consensus.cc:348] T df71b8fe94f84062bbf858f1cc4a0920 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.125751 5154 raft_consensus.cc:374] T df71b8fe94f84062bbf858f1cc4a0920 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.126119 5154 raft_consensus.cc:729] T df71b8fe94f84062bbf858f1cc4a0920 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b27c8ff42ea450bb5114ab97c1ed25d, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.130120 5155 ts_tablet_manager.cc:1428] T 778d8c794af64277b2f497708012e16c P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20260810 22:05:47.126857 5154 consensus_queue.cc:260] T df71b8fe94f84062bbf858f1cc4a0920 P 2b27c8ff42ea450bb5114ab97c1ed25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.131034 5155 tablet_bootstrap.cc:492] T f0037195580e420fad2a77957025fc74 P 2b27c8ff42ea450bb5114ab97c1ed25d: Bootstrap starting.
I20260810 22:05:47.131784 5156 tablet_bootstrap.cc:492] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd: No bootstrap required, opened a new log
I20260810 22:05:47.132278 5154 ts_tablet_manager.cc:1428] T df71b8fe94f84062bbf858f1cc4a0920 P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20260810 22:05:47.132411 5156 ts_tablet_manager.cc:1397] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd: Time spent bootstrapping tablet: real 0.048s user 0.012s sys 0.000s
I20260810 22:05:47.133258 5154 tablet_bootstrap.cc:492] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P 2b27c8ff42ea450bb5114ab97c1ed25d: Bootstrap starting.
I20260810 22:05:47.135942 5156 raft_consensus.cc:348] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.137005 5155 tablet_bootstrap.cc:654] T f0037195580e420fad2a77957025fc74 P 2b27c8ff42ea450bb5114ab97c1ed25d: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.142122 5156 raft_consensus.cc:374] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.142489 5156 raft_consensus.cc:729] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9235c169d85402faf437d72e9a38bdd, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.143334 5156 consensus_queue.cc:260] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.141834 5159 tablet_bootstrap.cc:654] T af1fb81326b54dbbb23c7b577bdaf751 P b9235c169d85402faf437d72e9a38bdd: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.149627 5156 ts_tablet_manager.cc:1428] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260810 22:05:47.150544 5156 tablet_bootstrap.cc:492] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd: Bootstrap starting.
W20260810 22:05:47.156105 4752 tablet.cc:2369] T 518b19decff74b1baf253900da181d46 P 2b27c8ff42ea450bb5114ab97c1ed25d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:05:47.156512 5154 tablet_bootstrap.cc:654] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P 2b27c8ff42ea450bb5114ab97c1ed25d: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.161967 5163 raft_consensus.cc:348] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.164094 5188 raft_consensus.cc:482] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:47.164155 5163 raft_consensus.cc:374] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.164556 5163 raft_consensus.cc:729] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552c4b7a11b4cb7a2fdaacbcdf219dc, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.164570 5188 raft_consensus.cc:504] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.166525 5188 leader_election.cc:288] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:47.165851 5163 consensus_queue.cc:260] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.170189 5156 tablet_bootstrap.cc:654] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.170301 4684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df71b8fe94f84062bbf858f1cc4a0920" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" is_pre_election: true
I20260810 22:05:47.171648 4684 raft_consensus.cc:2454] T df71b8fe94f84062bbf858f1cc4a0920 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9235c169d85402faf437d72e9a38bdd in term 0.
I20260810 22:05:47.174070 4781 leader_election.cc:302] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd; no voters:
I20260810 22:05:47.178082 5058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e619eaa340248da8a54a913c1d16b9f" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" is_pre_election: true
I20260810 22:05:47.180068 5188 raft_consensus.cc:2781] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:47.180526 5188 raft_consensus.cc:482] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:47.180908 5188 raft_consensus.cc:3037] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.185433 5059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df71b8fe94f84062bbf858f1cc4a0920" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" is_pre_election: true
W20260810 22:05:47.187387 4781 leader_election.cc:341] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:05:47.188519 4684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e619eaa340248da8a54a913c1d16b9f" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" is_pre_election: true
I20260810 22:05:47.190369 5156 tablet_bootstrap.cc:492] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd: No bootstrap required, opened a new log
W20260810 22:05:47.190428 4781 leader_election.cc:341] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:05:47.193312 4781 leader_election.cc:302] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9235c169d85402faf437d72e9a38bdd; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, a552c4b7a11b4cb7a2fdaacbcdf219dc
I20260810 22:05:47.194312 5154 tablet_bootstrap.cc:492] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P 2b27c8ff42ea450bb5114ab97c1ed25d: No bootstrap required, opened a new log
I20260810 22:05:47.194288 5182 raft_consensus.cc:2726] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 22:05:47.194818 5154 ts_tablet_manager.cc:1397] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent bootstrapping tablet: real 0.062s user 0.012s sys 0.003s
I20260810 22:05:47.195884 5156 ts_tablet_manager.cc:1397] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd: Time spent bootstrapping tablet: real 0.046s user 0.012s sys 0.003s
I20260810 22:05:47.167845 5166 raft_consensus.cc:348] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.196815 5166 raft_consensus.cc:374] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.195806 5188 raft_consensus.cc:504] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.197193 5166 raft_consensus.cc:729] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552c4b7a11b4cb7a2fdaacbcdf219dc, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.197613 5154 raft_consensus.cc:348] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } } peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } }
I20260810 22:05:47.198338 5154 raft_consensus.cc:374] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.198632 5154 raft_consensus.cc:729] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b27c8ff42ea450bb5114ab97c1ed25d, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.199381 5154 consensus_queue.cc:260] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P 2b27c8ff42ea450bb5114ab97c1ed25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } } peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } }
I20260810 22:05:47.200266 4684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df71b8fe94f84062bbf858f1cc4a0920" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d"
I20260810 22:05:47.200990 5188 leader_election.cc:288] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 election: Requested vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:47.202601 5154 ts_tablet_manager.cc:1428] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260810 22:05:47.203593 5154 tablet_bootstrap.cc:492] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d: Bootstrap starting.
I20260810 22:05:47.204389 5058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df71b8fe94f84062bbf858f1cc4a0920" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc"
I20260810 22:05:47.204787 5156 raft_consensus.cc:348] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.200953 4684 raft_consensus.cc:3037] T df71b8fe94f84062bbf858f1cc4a0920 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.207063 5156 raft_consensus.cc:374] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.207791 5156 raft_consensus.cc:729] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9235c169d85402faf437d72e9a38bdd, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.208945 5156 consensus_queue.cc:260] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.213753 5156 ts_tablet_manager.cc:1428] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260810 22:05:47.217325 5188 raft_consensus.cc:482] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:47.217883 5188 raft_consensus.cc:504] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } } peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } }
I20260810 22:05:47.218504 5154 tablet_bootstrap.cc:654] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.220016 5188 leader_election.cc:288] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793), 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093)
I20260810 22:05:47.217031 5156 tablet_bootstrap.cc:492] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd: Bootstrap starting.
I20260810 22:05:47.222147 4683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc4801ac5aeb4b4d8340ee2f5c0d73f6" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" is_pre_election: true
I20260810 22:05:47.223070 4683 raft_consensus.cc:2454] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9235c169d85402faf437d72e9a38bdd in term 0.
I20260810 22:05:47.224884 4781 leader_election.cc:302] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd; no voters:
I20260810 22:05:47.226282 5058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc4801ac5aeb4b4d8340ee2f5c0d73f6" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" is_pre_election: true
W20260810 22:05:47.228104 4781 leader_election.cc:341] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:05:47.231738 5188 raft_consensus.cc:2781] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:47.232110 5188 raft_consensus.cc:482] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:47.232450 5188 raft_consensus.cc:3037] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.235903 5156 tablet_bootstrap.cc:654] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.242326 5167 raft_consensus.cc:348] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.243350 5167 raft_consensus.cc:374] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.243705 5167 raft_consensus.cc:729] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552c4b7a11b4cb7a2fdaacbcdf219dc, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.244661 5167 consensus_queue.cc:260] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.198112 5166 consensus_queue.cc:260] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.250097 5059 raft_consensus.cc:2454] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9235c169d85402faf437d72e9a38bdd in term 0.
I20260810 22:05:47.254833 5156 tablet_bootstrap.cc:492] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd: No bootstrap required, opened a new log
I20260810 22:05:47.257009 5156 ts_tablet_manager.cc:1397] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd: Time spent bootstrapping tablet: real 0.040s user 0.013s sys 0.004s
I20260810 22:05:47.258200 5177 raft_consensus.cc:482] T 778d8c794af64277b2f497708012e16c P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:47.258672 5177 raft_consensus.cc:504] T 778d8c794af64277b2f497708012e16c P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.261778 5156 raft_consensus.cc:348] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.265630 5177 leader_election.cc:288] T 778d8c794af64277b2f497708012e16c P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:47.265748 5156 raft_consensus.cc:374] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.271452 5156 raft_consensus.cc:729] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9235c169d85402faf437d72e9a38bdd, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.271660 5182 raft_consensus.cc:482] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:47.272794 5182 raft_consensus.cc:504] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.275219 5182 leader_election.cc:288] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:47.277159 4683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518b19decff74b1baf253900da181d46" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" is_pre_election: true
I20260810 22:05:47.277948 4683 raft_consensus.cc:2454] T 518b19decff74b1baf253900da181d46 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9235c169d85402faf437d72e9a38bdd in term 0.
I20260810 22:05:47.279743 4781 leader_election.cc:302] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd; no voters:
I20260810 22:05:47.280354 5058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518b19decff74b1baf253900da181d46" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" is_pre_election: true
I20260810 22:05:47.280812 5182 raft_consensus.cc:2781] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:47.281185 5058 raft_consensus.cc:2454] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9235c169d85402faf437d72e9a38bdd in term 0.
I20260810 22:05:47.281387 5182 raft_consensus.cc:482] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:47.281850 5182 raft_consensus.cc:3037] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.282629 5185 raft_consensus.cc:482] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:47.283192 5185 raft_consensus.cc:504] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.285120 5185 leader_election.cc:288] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:47.287770 4683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628379a2077c4ea7896784b089a5ea0c" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" is_pre_election: true
I20260810 22:05:47.288331 5058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628379a2077c4ea7896784b089a5ea0c" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" is_pre_election: true
I20260810 22:05:47.288487 4683 raft_consensus.cc:2454] T 628379a2077c4ea7896784b089a5ea0c P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9235c169d85402faf437d72e9a38bdd in term 0.
I20260810 22:05:47.293876 5156 consensus_queue.cc:260] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.294513 4781 leader_election.cc:302] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd; no voters:
I20260810 22:05:47.298012 5185 raft_consensus.cc:2781] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:47.298442 5185 raft_consensus.cc:482] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:47.298885 5185 raft_consensus.cc:3037] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Advancing to term 1
W20260810 22:05:47.306110 4781 leader_election.cc:341] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:05:47.306393 5161 tablet_bootstrap.cc:492] T 416ba6bb2af24654a64eb1a23261254b P b9235c169d85402faf437d72e9a38bdd: No bootstrap required, opened a new log
I20260810 22:05:47.308301 5161 ts_tablet_manager.cc:1397] T 416ba6bb2af24654a64eb1a23261254b P b9235c169d85402faf437d72e9a38bdd: Time spent bootstrapping tablet: real 0.228s user 0.025s sys 0.053s
I20260810 22:05:47.311283 5123 heartbeater.cc:503] Master 127.27.17.190:44453 was elected leader, sending a full tablet report...
W20260810 22:05:47.313910 4940 tablet.cc:2369] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:05:47.296456 5182 raft_consensus.cc:504] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.315927 5216 raft_consensus.cc:482] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:47.316376 5216 raft_consensus.cc:504] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.314031 5185 raft_consensus.cc:504] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.319224 5185 leader_election.cc:288] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 election: Requested vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:47.319739 4683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518b19decff74b1baf253900da181d46" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d"
I20260810 22:05:47.320350 4683 raft_consensus.cc:3037] T 518b19decff74b1baf253900da181d46 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.320183 5161 raft_consensus.cc:348] T 416ba6bb2af24654a64eb1a23261254b P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.320870 5161 raft_consensus.cc:374] T 416ba6bb2af24654a64eb1a23261254b P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.322867 5161 raft_consensus.cc:729] T 416ba6bb2af24654a64eb1a23261254b P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9235c169d85402faf437d72e9a38bdd, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.322821 4681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628379a2077c4ea7896784b089a5ea0c" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d"
I20260810 22:05:47.323462 4681 raft_consensus.cc:3037] T 628379a2077c4ea7896784b089a5ea0c P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.323573 5161 consensus_queue.cc:260] T 416ba6bb2af24654a64eb1a23261254b P b9235c169d85402faf437d72e9a38bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.323619 5182 leader_election.cc:288] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 election: Requested vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:47.332098 5161 ts_tablet_manager.cc:1428] T 416ba6bb2af24654a64eb1a23261254b P b9235c169d85402faf437d72e9a38bdd: Time spent starting tablet: real 0.023s user 0.005s sys 0.002s
I20260810 22:05:47.333141 5161 tablet_bootstrap.cc:492] T e33e69aaf4a64216b2f995a9e9e9e1e2 P b9235c169d85402faf437d72e9a38bdd: Bootstrap starting.
I20260810 22:05:47.348352 5153 tablet_bootstrap.cc:492] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d: No bootstrap required, opened a new log
I20260810 22:05:47.348866 5153 ts_tablet_manager.cc:1397] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent bootstrapping tablet: real 0.281s user 0.042s sys 0.037s
I20260810 22:05:47.351560 5153 raft_consensus.cc:348] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.352337 5153 raft_consensus.cc:374] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.352408 5161 tablet_bootstrap.cc:654] T e33e69aaf4a64216b2f995a9e9e9e1e2 P b9235c169d85402faf437d72e9a38bdd: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.352638 5153 raft_consensus.cc:729] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b27c8ff42ea450bb5114ab97c1ed25d, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.356659 5216 leader_election.cc:288] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
I20260810 22:05:47.358507 5185 raft_consensus.cc:482] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:47.359025 5185 raft_consensus.cc:504] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.362772 4680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0037195580e420fad2a77957025fc74" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" is_pre_election: true
W20260810 22:05:47.364555 4781 leader_election.cc:341] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:05:47.365928 5185 leader_election.cc:288] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:47.367079 5177 raft_consensus.cc:482] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:47.367599 5177 raft_consensus.cc:504] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.370031 5177 leader_election.cc:288] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:47.372406 5166 ts_tablet_manager.cc:1428] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent starting tablet: real 0.264s user 0.017s sys 0.002s
I20260810 22:05:47.373625 5166 tablet_bootstrap.cc:492] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Bootstrap starting.
I20260810 22:05:47.371812 5153 consensus_queue.cc:260] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.376856 5161 tablet_bootstrap.cc:492] T e33e69aaf4a64216b2f995a9e9e9e1e2 P b9235c169d85402faf437d72e9a38bdd: No bootstrap required, opened a new log
I20260810 22:05:47.377054 5153 ts_tablet_manager.cc:1428] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent starting tablet: real 0.028s user 0.006s sys 0.002s
I20260810 22:05:47.377416 5161 ts_tablet_manager.cc:1397] T e33e69aaf4a64216b2f995a9e9e9e1e2 P b9235c169d85402faf437d72e9a38bdd: Time spent bootstrapping tablet: real 0.044s user 0.010s sys 0.005s
I20260810 22:05:47.385829 5177 raft_consensus.cc:482] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:47.386284 5177 raft_consensus.cc:504] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.388140 5177 leader_election.cc:288] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:47.380686 5161 raft_consensus.cc:348] T e33e69aaf4a64216b2f995a9e9e9e1e2 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.394991 5161 raft_consensus.cc:374] T e33e69aaf4a64216b2f995a9e9e9e1e2 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.396550 5161 raft_consensus.cc:729] T e33e69aaf4a64216b2f995a9e9e9e1e2 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9235c169d85402faf437d72e9a38bdd, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.396857 4681 raft_consensus.cc:2454] T 628379a2077c4ea7896784b089a5ea0c P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9235c169d85402faf437d72e9a38bdd in term 1.
I20260810 22:05:47.397459 5161 consensus_queue.cc:260] T e33e69aaf4a64216b2f995a9e9e9e1e2 P b9235c169d85402faf437d72e9a38bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.398280 4781 leader_election.cc:302] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd; no voters:
I20260810 22:05:47.399205 5185 raft_consensus.cc:2781] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:47.399295 4683 raft_consensus.cc:2454] T 518b19decff74b1baf253900da181d46 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9235c169d85402faf437d72e9a38bdd in term 1.
I20260810 22:05:47.399686 5185 raft_consensus.cc:686] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [term 1 LEADER]: Becoming Leader. State: Replica: b9235c169d85402faf437d72e9a38bdd, State: Running, Role: LEADER
I20260810 22:05:47.400069 5161 ts_tablet_manager.cc:1428] T e33e69aaf4a64216b2f995a9e9e9e1e2 P b9235c169d85402faf437d72e9a38bdd: Time spent starting tablet: real 0.022s user 0.004s sys 0.002s
I20260810 22:05:47.400277 4781 leader_election.cc:302] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd; no voters:
I20260810 22:05:47.400931 5161 tablet_bootstrap.cc:492] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd: Bootstrap starting.
I20260810 22:05:47.400625 5185 consensus_queue.cc:237] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.403450 4684 raft_consensus.cc:2454] T df71b8fe94f84062bbf858f1cc4a0920 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9235c169d85402faf437d72e9a38bdd in term 1.
I20260810 22:05:47.401731 5182 raft_consensus.cc:2781] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:47.405704 5182 raft_consensus.cc:686] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd [term 1 LEADER]: Becoming Leader. State: Replica: b9235c169d85402faf437d72e9a38bdd, State: Running, Role: LEADER
I20260810 22:05:47.406610 5182 consensus_queue.cc:237] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.410864 5188 raft_consensus.cc:504] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } } peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } }
I20260810 22:05:47.412324 4781 leader_election.cc:302] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd; no voters: a552c4b7a11b4cb7a2fdaacbcdf219dc
I20260810 22:05:47.412555 5188 leader_election.cc:288] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 election: Requested vote from peers a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793), 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093)
I20260810 22:05:47.412814 5154 tablet_bootstrap.cc:492] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d: No bootstrap required, opened a new log
I20260810 22:05:47.413249 5154 ts_tablet_manager.cc:1397] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent bootstrapping tablet: real 0.210s user 0.035s sys 0.046s
I20260810 22:05:47.414912 5188 raft_consensus.cc:2781] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:47.415408 5188 raft_consensus.cc:686] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd [term 1 LEADER]: Becoming Leader. State: Replica: b9235c169d85402faf437d72e9a38bdd, State: Running, Role: LEADER
I20260810 22:05:47.416010 5155 tablet_bootstrap.cc:492] T f0037195580e420fad2a77957025fc74 P 2b27c8ff42ea450bb5114ab97c1ed25d: No bootstrap required, opened a new log
I20260810 22:05:47.416579 5155 ts_tablet_manager.cc:1397] T f0037195580e420fad2a77957025fc74 P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent bootstrapping tablet: real 0.286s user 0.029s sys 0.049s
I20260810 22:05:47.416137 5154 raft_consensus.cc:348] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.417207 5154 raft_consensus.cc:374] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.417634 5154 raft_consensus.cc:729] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b27c8ff42ea450bb5114ab97c1ed25d, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.418820 5154 consensus_queue.cc:260] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.419760 5155 raft_consensus.cc:348] T f0037195580e420fad2a77957025fc74 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.420591 5155 raft_consensus.cc:374] T f0037195580e420fad2a77957025fc74 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.420914 5155 raft_consensus.cc:729] T f0037195580e420fad2a77957025fc74 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b27c8ff42ea450bb5114ab97c1ed25d, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.422650 5154 ts_tablet_manager.cc:1428] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20260810 22:05:47.430876 5155 consensus_queue.cc:260] T f0037195580e420fad2a77957025fc74 P 2b27c8ff42ea450bb5114ab97c1ed25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.437750 5167 ts_tablet_manager.cc:1428] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent starting tablet: real 0.339s user 0.017s sys 0.004s
I20260810 22:05:47.439795 5167 tablet_bootstrap.cc:492] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc: Bootstrap starting.
I20260810 22:05:47.442916 5166 tablet_bootstrap.cc:654] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.445103 5156 ts_tablet_manager.cc:1428] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd: Time spent starting tablet: real 0.187s user 0.025s sys 0.030s
I20260810 22:05:47.438223 5163 ts_tablet_manager.cc:1428] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent starting tablet: real 0.374s user 0.019s sys 0.008s
I20260810 22:05:47.433293 5155 ts_tablet_manager.cc:1428] T f0037195580e420fad2a77957025fc74 P 2b27c8ff42ea450bb5114ab97c1ed25d: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260810 22:05:47.451120 5161 tablet_bootstrap.cc:654] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.447053 5188 consensus_queue.cc:237] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.477551 5159 tablet_bootstrap.cc:492] T af1fb81326b54dbbb23c7b577bdaf751 P b9235c169d85402faf437d72e9a38bdd: No bootstrap required, opened a new log
I20260810 22:05:47.478016 5159 ts_tablet_manager.cc:1397] T af1fb81326b54dbbb23c7b577bdaf751 P b9235c169d85402faf437d72e9a38bdd: Time spent bootstrapping tablet: real 0.389s user 0.043s sys 0.076s
I20260810 22:05:47.480476 5159 raft_consensus.cc:348] T af1fb81326b54dbbb23c7b577bdaf751 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.481242 5159 raft_consensus.cc:374] T af1fb81326b54dbbb23c7b577bdaf751 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.481575 5159 raft_consensus.cc:729] T af1fb81326b54dbbb23c7b577bdaf751 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9235c169d85402faf437d72e9a38bdd, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.482191 5159 consensus_queue.cc:260] T af1fb81326b54dbbb23c7b577bdaf751 P b9235c169d85402faf437d72e9a38bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.484319 5159 ts_tablet_manager.cc:1428] T af1fb81326b54dbbb23c7b577bdaf751 P b9235c169d85402faf437d72e9a38bdd: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260810 22:05:47.491006 4874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778d8c794af64277b2f497708012e16c" candidate_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9235c169d85402faf437d72e9a38bdd" is_pre_election: true
I20260810 22:05:47.493498 4874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1fb81326b54dbbb23c7b577bdaf751" candidate_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9235c169d85402faf437d72e9a38bdd" is_pre_election: true
I20260810 22:05:47.494098 4874 raft_consensus.cc:2454] T af1fb81326b54dbbb23c7b577bdaf751 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b27c8ff42ea450bb5114ab97c1ed25d in term 0.
I20260810 22:05:47.495704 4874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33e69aaf4a64216b2f995a9e9e9e1e2" candidate_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9235c169d85402faf437d72e9a38bdd" is_pre_election: true
I20260810 22:05:47.496805 4874 raft_consensus.cc:2454] T e33e69aaf4a64216b2f995a9e9e9e1e2 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b27c8ff42ea450bb5114ab97c1ed25d in term 0.
W20260810 22:05:47.497896 4595 leader_election.cc:341] T 778d8c794af64277b2f497708012e16c P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:05:47.499620 4595 leader_election.cc:302] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd; no voters:
I20260810 22:05:47.500787 5177 raft_consensus.cc:2781] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:47.501222 5177 raft_consensus.cc:482] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:47.501622 5177 raft_consensus.cc:3037] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.510838 5177 raft_consensus.cc:504] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.506016 4470 catalog_manager.cc:5495] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd reported cstate change: term changed from 0 to 1, leader changed from <none> to b9235c169d85402faf437d72e9a38bdd (127.27.17.130). New cstate: current_term: 1 leader_uuid: "b9235c169d85402faf437d72e9a38bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } health_report { overall_health: UNKNOWN } } }
I20260810 22:05:47.513062 5161 tablet_bootstrap.cc:492] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd: No bootstrap required, opened a new log
I20260810 22:05:47.515262 4595 leader_election.cc:302] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd; no voters:
I20260810 22:05:47.514942 5161 ts_tablet_manager.cc:1397] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd: Time spent bootstrapping tablet: real 0.114s user 0.013s sys 0.007s
I20260810 22:05:47.516528 5176 raft_consensus.cc:2781] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:47.516911 5176 raft_consensus.cc:482] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:47.517279 5176 raft_consensus.cc:3037] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Advancing to term 1
W20260810 22:05:47.504395 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.17.129:43093, user_credentials={real_user=slave}} blocked reactor thread for 91965.3us
I20260810 22:05:47.526538 5176 raft_consensus.cc:504] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.531016 4874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1fb81326b54dbbb23c7b577bdaf751" candidate_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9235c169d85402faf437d72e9a38bdd"
I20260810 22:05:47.531653 4874 raft_consensus.cc:3037] T af1fb81326b54dbbb23c7b577bdaf751 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.532286 4684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc4801ac5aeb4b4d8340ee2f5c0d73f6" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d"
I20260810 22:05:47.532805 4684 raft_consensus.cc:3037] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.539261 4684 raft_consensus.cc:2454] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9235c169d85402faf437d72e9a38bdd in term 1.
I20260810 22:05:47.547186 4872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33e69aaf4a64216b2f995a9e9e9e1e2" candidate_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9235c169d85402faf437d72e9a38bdd"
I20260810 22:05:47.548205 4872 raft_consensus.cc:3037] T e33e69aaf4a64216b2f995a9e9e9e1e2 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.555156 4872 raft_consensus.cc:2454] T e33e69aaf4a64216b2f995a9e9e9e1e2 P b9235c169d85402faf437d72e9a38bdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b27c8ff42ea450bb5114ab97c1ed25d in term 1.
I20260810 22:05:47.561915 4595 leader_election.cc:302] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd; no voters:
I20260810 22:05:47.563776 5161 raft_consensus.cc:348] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.568025 4781 leader_election.cc:302] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd; no voters:
I20260810 22:05:47.573607 5188 raft_consensus.cc:2781] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:47.574100 5188 raft_consensus.cc:686] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [term 1 LEADER]: Becoming Leader. State: Replica: b9235c169d85402faf437d72e9a38bdd, State: Running, Role: LEADER
I20260810 22:05:47.574941 5188 consensus_queue.cc:237] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } } peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } }
I20260810 22:05:47.578285 5176 leader_election.cc:288] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 election: Requested vote from peers b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:47.579382 5176 raft_consensus.cc:2781] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:47.580066 5176 raft_consensus.cc:686] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 LEADER]: Becoming Leader. State: Replica: 2b27c8ff42ea450bb5114ab97c1ed25d, State: Running, Role: LEADER
I20260810 22:05:47.581316 5176 consensus_queue.cc:237] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
W20260810 22:05:47.582787 5125 tablet.cc:2369] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:05:47.582965 5059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628379a2077c4ea7896784b089a5ea0c" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc"
I20260810 22:05:47.584270 5058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518b19decff74b1baf253900da181d46" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc"
I20260810 22:05:47.584909 5058 raft_consensus.cc:3037] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.585482 5163 tablet_bootstrap.cc:492] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc: Bootstrap starting.
I20260810 22:05:47.572484 5161 raft_consensus.cc:374] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.587031 5161 raft_consensus.cc:729] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9235c169d85402faf437d72e9a38bdd, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.587726 5161 consensus_queue.cc:260] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.588863 5167 tablet_bootstrap.cc:700] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent opening tablet: real 0.136s user 0.007s sys 0.032s
I20260810 22:05:47.589526 5167 tablet_bootstrap.cc:654] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.592279 5161 ts_tablet_manager.cc:1428] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd: Time spent starting tablet: real 0.076s user 0.008s sys 0.000s
I20260810 22:05:47.594422 4874 raft_consensus.cc:2454] T af1fb81326b54dbbb23c7b577bdaf751 P b9235c169d85402faf437d72e9a38bdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b27c8ff42ea450bb5114ab97c1ed25d in term 1.
I20260810 22:05:47.602286 4470 catalog_manager.cc:5495] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd reported cstate change: term changed from 0 to 1, leader changed from <none> to b9235c169d85402faf437d72e9a38bdd (127.27.17.130). New cstate: current_term: 1 leader_uuid: "b9235c169d85402faf437d72e9a38bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } health_report { overall_health: UNKNOWN } } }
I20260810 22:05:47.604133 4470 catalog_manager.cc:5495] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd reported cstate change: term changed from 0 to 1, leader changed from <none> to b9235c169d85402faf437d72e9a38bdd (127.27.17.130). New cstate: current_term: 1 leader_uuid: "b9235c169d85402faf437d72e9a38bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } health_report { overall_health: UNKNOWN } } }
I20260810 22:05:47.590595 5177 leader_election.cc:288] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 election: Requested vote from peers b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:47.595759 5243 raft_consensus.cc:482] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:47.607873 4595 leader_election.cc:302] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd; no voters:
I20260810 22:05:47.609169 5177 raft_consensus.cc:2781] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:47.608886 5243 raft_consensus.cc:504] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.610950 5177 raft_consensus.cc:686] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 LEADER]: Becoming Leader. State: Replica: 2b27c8ff42ea450bb5114ab97c1ed25d, State: Running, Role: LEADER
I20260810 22:05:47.611476 5243 leader_election.cc:288] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:47.611775 5177 consensus_queue.cc:237] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.618007 4874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416ba6bb2af24654a64eb1a23261254b" candidate_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9235c169d85402faf437d72e9a38bdd" is_pre_election: true
I20260810 22:05:47.618628 4874 raft_consensus.cc:2454] T 416ba6bb2af24654a64eb1a23261254b P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b27c8ff42ea450bb5114ab97c1ed25d in term 0.
I20260810 22:05:47.619493 4595 leader_election.cc:302] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd; no voters:
I20260810 22:05:47.620260 5177 raft_consensus.cc:2781] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:47.620694 5177 raft_consensus.cc:482] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:47.621094 5177 raft_consensus.cc:3037] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.632658 4472 catalog_manager.cc:5495] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129). New cstate: current_term: 1 leader_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } health_report { overall_health: UNKNOWN } } }
I20260810 22:05:47.640780 5163 tablet_bootstrap.cc:654] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.651846 5058 raft_consensus.cc:2454] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9235c169d85402faf437d72e9a38bdd in term 1.
I20260810 22:05:47.653754 5059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc4801ac5aeb4b4d8340ee2f5c0d73f6" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc"
W20260810 22:05:47.654024 4781 leader_election.cc:341] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:05:47.653976 5177 raft_consensus.cc:504] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.653407 5058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0037195580e420fad2a77957025fc74" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" is_pre_election: true
W20260810 22:05:47.655334 4781 leader_election.cc:341] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:05:47.655836 5058 raft_consensus.cc:2454] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9235c169d85402faf437d72e9a38bdd in term 0.
I20260810 22:05:47.655975 5177 leader_election.cc:288] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 election: Requested vote from peers b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:47.657279 4874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416ba6bb2af24654a64eb1a23261254b" candidate_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9235c169d85402faf437d72e9a38bdd"
I20260810 22:05:47.657491 4781 leader_election.cc:302] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc, b9235c169d85402faf437d72e9a38bdd; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d
I20260810 22:05:47.657979 4874 raft_consensus.cc:3037] T 416ba6bb2af24654a64eb1a23261254b P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.658529 5188 raft_consensus.cc:2781] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:47.658864 5188 raft_consensus.cc:482] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:47.658464 5059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1fb81326b54dbbb23c7b577bdaf751" candidate_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" is_pre_election: true
I20260810 22:05:47.659188 5188 raft_consensus.cc:3037] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.665612 5188 raft_consensus.cc:504] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.668491 4684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0037195580e420fad2a77957025fc74" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d"
I20260810 22:05:47.669239 4684 raft_consensus.cc:3037] T f0037195580e420fad2a77957025fc74 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.658659 5058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778d8c794af64277b2f497708012e16c" candidate_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" is_pre_election: true
I20260810 22:05:47.672719 4874 raft_consensus.cc:2454] T 416ba6bb2af24654a64eb1a23261254b P b9235c169d85402faf437d72e9a38bdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b27c8ff42ea450bb5114ab97c1ed25d in term 1.
I20260810 22:05:47.675163 4595 leader_election.cc:302] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd; no voters:
I20260810 22:05:47.675583 5188 leader_election.cc:288] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 election: Requested vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:47.677066 4684 raft_consensus.cc:2454] T f0037195580e420fad2a77957025fc74 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9235c169d85402faf437d72e9a38bdd in term 1.
I20260810 22:05:47.678385 4781 leader_election.cc:302] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd; no voters:
I20260810 22:05:47.678417 4470 catalog_manager.cc:5495] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd reported cstate change: term changed from 0 to 1, leader changed from <none> to b9235c169d85402faf437d72e9a38bdd (127.27.17.130). New cstate: current_term: 1 leader_uuid: "b9235c169d85402faf437d72e9a38bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } health_report { overall_health: UNKNOWN } } }
I20260810 22:05:47.679395 5188 raft_consensus.cc:2781] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:47.680140 5188 raft_consensus.cc:686] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [term 1 LEADER]: Becoming Leader. State: Replica: b9235c169d85402faf437d72e9a38bdd, State: Running, Role: LEADER
I20260810 22:05:47.681134 5188 consensus_queue.cc:237] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.684527 5177 raft_consensus.cc:2781] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:47.685498 5177 raft_consensus.cc:686] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 LEADER]: Becoming Leader. State: Replica: 2b27c8ff42ea450bb5114ab97c1ed25d, State: Running, Role: LEADER
I20260810 22:05:47.686398 5177 consensus_queue.cc:237] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.661872 5057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33e69aaf4a64216b2f995a9e9e9e1e2" candidate_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" is_pre_election: true
I20260810 22:05:47.671275 5056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1fb81326b54dbbb23c7b577bdaf751" candidate_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc"
I20260810 22:05:47.694809 4468 catalog_manager.cc:5495] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd reported cstate change: term changed from 0 to 1, leader changed from <none> to b9235c169d85402faf437d72e9a38bdd (127.27.17.130). New cstate: current_term: 1 leader_uuid: "b9235c169d85402faf437d72e9a38bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } health_report { overall_health: UNKNOWN } } }
I20260810 22:05:47.697551 5176 raft_consensus.cc:482] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:47.697989 5176 raft_consensus.cc:504] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.699908 5176 leader_election.cc:288] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
W20260810 22:05:47.701067 4591 leader_election.cc:341] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:05:47.701761 5056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416ba6bb2af24654a64eb1a23261254b" candidate_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" is_pre_election: true
W20260810 22:05:47.702129 4591 leader_election.cc:341] T 778d8c794af64277b2f497708012e16c P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:05:47.702468 4591 leader_election.cc:302] T 778d8c794af64277b2f497708012e16c P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b27c8ff42ea450bb5114ab97c1ed25d; no voters: a552c4b7a11b4cb7a2fdaacbcdf219dc, b9235c169d85402faf437d72e9a38bdd
I20260810 22:05:47.702709 5057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416ba6bb2af24654a64eb1a23261254b" candidate_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc"
W20260810 22:05:47.703506 4591 leader_election.cc:341] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793): Illegal state: must be running to vote when last-logged opid is not known
W20260810 22:05:47.704298 4591 leader_election.cc:341] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:05:47.706604 4874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e619eaa340248da8a54a913c1d16b9f" candidate_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9235c169d85402faf437d72e9a38bdd" is_pre_election: true
I20260810 22:05:47.707201 4874 raft_consensus.cc:2454] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b27c8ff42ea450bb5114ab97c1ed25d in term 0.
I20260810 22:05:47.708312 4595 leader_election.cc:302] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd; no voters:
I20260810 22:05:47.709074 5243 raft_consensus.cc:2726] T 778d8c794af64277b2f497708012e16c P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 22:05:47.709645 5243 raft_consensus.cc:2781] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:47.709936 5243 raft_consensus.cc:482] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:47.710216 5243 raft_consensus.cc:3037] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.711683 5058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0037195580e420fad2a77957025fc74" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc"
I20260810 22:05:47.712222 5058 raft_consensus.cc:3037] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.714273 5057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e619eaa340248da8a54a913c1d16b9f" candidate_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" is_pre_election: true
I20260810 22:05:47.717736 5059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33e69aaf4a64216b2f995a9e9e9e1e2" candidate_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc"
W20260810 22:05:47.705113 4591 leader_election.cc:341] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793): Illegal state: must be running to vote when last-logged opid is not known
W20260810 22:05:47.720607 4591 leader_election.cc:341] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793): Illegal state: must be running to vote when last-logged opid is not known
W20260810 22:05:47.721446 4591 leader_election.cc:341] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793): Illegal state: must be running to vote when last-logged opid is not known
W20260810 22:05:47.722255 4591 leader_election.cc:341] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:05:47.716921 5243 raft_consensus.cc:504] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.723955 4684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0037195580e420fad2a77957025fc74" candidate_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" is_pre_election: true
I20260810 22:05:47.724750 4684 raft_consensus.cc:2379] T f0037195580e420fad2a77957025fc74 P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a552c4b7a11b4cb7a2fdaacbcdf219dc in current term 1: Already voted for candidate b9235c169d85402faf437d72e9a38bdd in this term.
I20260810 22:05:47.726390 4874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e619eaa340248da8a54a913c1d16b9f" candidate_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9235c169d85402faf437d72e9a38bdd"
I20260810 22:05:47.726981 4874 raft_consensus.cc:3037] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.733675 4874 raft_consensus.cc:2454] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b27c8ff42ea450bb5114ab97c1ed25d in term 1.
I20260810 22:05:47.734732 4595 leader_election.cc:302] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd; no voters:
I20260810 22:05:47.734941 4872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0037195580e420fad2a77957025fc74" candidate_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9235c169d85402faf437d72e9a38bdd" is_pre_election: true
I20260810 22:05:47.735601 5176 raft_consensus.cc:2781] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:47.736027 5176 raft_consensus.cc:686] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 LEADER]: Becoming Leader. State: Replica: 2b27c8ff42ea450bb5114ab97c1ed25d, State: Running, Role: LEADER
I20260810 22:05:47.736649 4974 leader_election.cc:302] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:05:47.736814 5176 consensus_queue.cc:237] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.744220 5058 raft_consensus.cc:2454] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9235c169d85402faf437d72e9a38bdd in term 1.
I20260810 22:05:47.745615 5167 tablet_bootstrap.cc:492] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc: No bootstrap required, opened a new log
I20260810 22:05:47.746129 5167 ts_tablet_manager.cc:1397] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent bootstrapping tablet: real 0.307s user 0.017s sys 0.039s
I20260810 22:05:47.746078 5220 raft_consensus.cc:2726] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 22:05:47.746485 5243 leader_election.cc:288] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 election: Requested vote from peers b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:47.749430 5059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e619eaa340248da8a54a913c1d16b9f" candidate_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc"
I20260810 22:05:47.749873 5166 tablet_bootstrap.cc:492] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc: No bootstrap required, opened a new log
I20260810 22:05:47.745616 5163 tablet_bootstrap.cc:492] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc: No bootstrap required, opened a new log
I20260810 22:05:47.750248 5166 ts_tablet_manager.cc:1397] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent bootstrapping tablet: real 0.377s user 0.024s sys 0.047s
I20260810 22:05:47.750434 5163 ts_tablet_manager.cc:1397] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent bootstrapping tablet: real 0.165s user 0.026s sys 0.029s
I20260810 22:05:47.752549 5166 raft_consensus.cc:348] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.753197 5166 raft_consensus.cc:374] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.753825 5166 raft_consensus.cc:729] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552c4b7a11b4cb7a2fdaacbcdf219dc, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.754662 5166 consensus_queue.cc:260] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
W20260810 22:05:47.757652 4591 leader_election.cc:341] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793): Illegal state: must be running to vote when last-logged opid is not known
I20260810 22:05:47.758912 5166 ts_tablet_manager.cc:1428] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260810 22:05:47.759840 5166 tablet_bootstrap.cc:492] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Bootstrap starting.
I20260810 22:05:47.759595 5167 raft_consensus.cc:348] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.760325 5167 raft_consensus.cc:374] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.760612 5167 raft_consensus.cc:729] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552c4b7a11b4cb7a2fdaacbcdf219dc, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.761286 5167 consensus_queue.cc:260] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.763494 5167 ts_tablet_manager.cc:1428] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260810 22:05:47.764318 5167 tablet_bootstrap.cc:492] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc: Bootstrap starting.
I20260810 22:05:47.770496 5167 tablet_bootstrap.cc:654] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.775208 5166 tablet_bootstrap.cc:654] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.783044 4470 catalog_manager.cc:5495] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129). New cstate: current_term: 1 leader_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } health_report { overall_health: UNKNOWN } } }
I20260810 22:05:47.752724 5163 raft_consensus.cc:348] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.787214 5163 raft_consensus.cc:374] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.787894 5163 raft_consensus.cc:729] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552c4b7a11b4cb7a2fdaacbcdf219dc, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.785712 4470 catalog_manager.cc:5495] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129). New cstate: current_term: 1 leader_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } health_report { overall_health: UNKNOWN } } }
I20260810 22:05:47.788690 5163 consensus_queue.cc:260] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.789597 4470 catalog_manager.cc:5495] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129). New cstate: current_term: 1 leader_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } health_report { overall_health: UNKNOWN } } }
I20260810 22:05:47.790483 5167 tablet_bootstrap.cc:492] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc: No bootstrap required, opened a new log
I20260810 22:05:47.790887 5167 ts_tablet_manager.cc:1397] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.010s
I20260810 22:05:47.793823 5163 ts_tablet_manager.cc:1428] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent starting tablet: real 0.043s user 0.007s sys 0.000s
I20260810 22:05:47.794827 5163 tablet_bootstrap.cc:492] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Bootstrap starting.
I20260810 22:05:47.801407 5167 raft_consensus.cc:348] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.802215 5167 raft_consensus.cc:374] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.802551 5167 raft_consensus.cc:729] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552c4b7a11b4cb7a2fdaacbcdf219dc, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.803267 5167 consensus_queue.cc:260] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.804924 5163 tablet_bootstrap.cc:654] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.805425 5166 tablet_bootstrap.cc:492] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc: No bootstrap required, opened a new log
I20260810 22:05:47.805912 5166 ts_tablet_manager.cc:1397] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent bootstrapping tablet: real 0.046s user 0.007s sys 0.005s
I20260810 22:05:47.806387 5167 ts_tablet_manager.cc:1428] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260810 22:05:47.807260 5167 tablet_bootstrap.cc:492] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc: Bootstrap starting.
I20260810 22:05:47.808563 5166 raft_consensus.cc:348] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.813457 5166 raft_consensus.cc:374] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.813876 5166 raft_consensus.cc:729] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552c4b7a11b4cb7a2fdaacbcdf219dc, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.817503 5188 consensus_queue.cc:1035] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:47.815236 5220 raft_consensus.cc:482] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:47.818475 5220 raft_consensus.cc:504] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.818724 5166 consensus_queue.cc:260] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.826646 4683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628379a2077c4ea7896784b089a5ea0c" candidate_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" is_pre_election: true
I20260810 22:05:47.827342 5166 ts_tablet_manager.cc:1428] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent starting tablet: real 0.021s user 0.005s sys 0.002s
I20260810 22:05:47.827469 4683 raft_consensus.cc:2379] T 628379a2077c4ea7896784b089a5ea0c P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a552c4b7a11b4cb7a2fdaacbcdf219dc in current term 1: Already voted for candidate b9235c169d85402faf437d72e9a38bdd in this term.
I20260810 22:05:47.828135 4874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628379a2077c4ea7896784b089a5ea0c" candidate_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9235c169d85402faf437d72e9a38bdd" is_pre_election: true
I20260810 22:05:47.830204 4974 leader_election.cc:302] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:05:47.813088 5167 tablet_bootstrap.cc:654] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc: Neither blocks nor log segments found. Creating new log.
I20260810 22:05:47.831514 5216 raft_consensus.cc:3037] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.835947 5220 leader_election.cc:288] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
I20260810 22:05:47.841223 5216 raft_consensus.cc:2726] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 22:05:47.861681 5216 raft_consensus.cc:482] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:47.862274 5216 raft_consensus.cc:504] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.865561 4683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1fb81326b54dbbb23c7b577bdaf751" candidate_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" is_pre_election: true
I20260810 22:05:47.866204 4874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1fb81326b54dbbb23c7b577bdaf751" candidate_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9235c169d85402faf437d72e9a38bdd" is_pre_election: true
I20260810 22:05:47.867275 4874 raft_consensus.cc:2379] T af1fb81326b54dbbb23c7b577bdaf751 P b9235c169d85402faf437d72e9a38bdd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a552c4b7a11b4cb7a2fdaacbcdf219dc in current term 1: Already voted for candidate 2b27c8ff42ea450bb5114ab97c1ed25d in this term.
I20260810 22:05:47.868635 4974 leader_election.cc:302] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:05:47.869807 5220 raft_consensus.cc:3037] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:47.879669 5216 leader_election.cc:288] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
I20260810 22:05:47.883832 5220 raft_consensus.cc:2726] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 22:05:47.884922 5188 consensus_queue.cc:1035] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:47.893514 5167 tablet_bootstrap.cc:492] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc: No bootstrap required, opened a new log
I20260810 22:05:47.893972 5167 ts_tablet_manager.cc:1397] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent bootstrapping tablet: real 0.087s user 0.012s sys 0.014s
I20260810 22:05:47.895949 5163 tablet_bootstrap.cc:492] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc: No bootstrap required, opened a new log
I20260810 22:05:47.896829 5163 ts_tablet_manager.cc:1397] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent bootstrapping tablet: real 0.102s user 0.035s sys 0.049s
I20260810 22:05:47.896975 5167 raft_consensus.cc:348] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.897744 5167 raft_consensus.cc:374] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.898054 5167 raft_consensus.cc:729] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552c4b7a11b4cb7a2fdaacbcdf219dc, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.898867 5167 consensus_queue.cc:260] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.902434 5163 raft_consensus.cc:348] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } } peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } }
I20260810 22:05:47.903116 5163 raft_consensus.cc:374] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:05:47.903376 5163 raft_consensus.cc:729] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552c4b7a11b4cb7a2fdaacbcdf219dc, State: Initialized, Role: FOLLOWER
I20260810 22:05:47.903976 5163 consensus_queue.cc:260] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } } peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } }
I20260810 22:05:47.905697 5163 ts_tablet_manager.cc:1428] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20260810 22:05:47.907850 5167 ts_tablet_manager.cc:1428] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc: Time spent starting tablet: real 0.014s user 0.004s sys 0.002s
I20260810 22:05:47.928691 5223 consensus_queue.cc:1035] T 628379a2077c4ea7896784b089a5ea0c P b9235c169d85402faf437d72e9a38bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 22:05:47.930765 5227 consensus_queue.cc:1035] T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 22:05:47.996577 5188 raft_consensus.cc:482] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:47.997008 5188 raft_consensus.cc:504] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:47.998814 5188 leader_election.cc:288] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:47.999068 5176 consensus_queue.cc:1035] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:48.000020 5058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778d8c794af64277b2f497708012e16c" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" is_pre_election: true
I20260810 22:05:48.000398 4684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778d8c794af64277b2f497708012e16c" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" is_pre_election: true
I20260810 22:05:48.000629 5058 raft_consensus.cc:2454] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9235c169d85402faf437d72e9a38bdd in term 0.
I20260810 22:05:48.001196 4684 raft_consensus.cc:2454] T 778d8c794af64277b2f497708012e16c P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9235c169d85402faf437d72e9a38bdd in term 0.
I20260810 22:05:48.001708 4781 leader_election.cc:302] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc, b9235c169d85402faf437d72e9a38bdd; no voters:
I20260810 22:05:48.002832 5227 raft_consensus.cc:2781] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:05:48.003221 5227 raft_consensus.cc:482] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:05:48.003604 5227 raft_consensus.cc:3037] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:48.004068 5058 raft_consensus.cc:3037] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:48.011385 5188 consensus_queue.cc:1035] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:48.014097 5227 raft_consensus.cc:504] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:48.026890 5176 consensus_queue.cc:1035] T af1fb81326b54dbbb23c7b577bdaf751 P 2b27c8ff42ea450bb5114ab97c1ed25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:48.023013 4684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778d8c794af64277b2f497708012e16c" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d"
I20260810 22:05:48.028820 4684 raft_consensus.cc:3037] T 778d8c794af64277b2f497708012e16c P 2b27c8ff42ea450bb5114ab97c1ed25d [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:48.035712 5227 leader_election.cc:288] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 election: Requested vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), a552c4b7a11b4cb7a2fdaacbcdf219dc (127.27.17.131:44793)
I20260810 22:05:48.036372 5223 consensus_queue.cc:1035] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:48.038431 5059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778d8c794af64277b2f497708012e16c" candidate_uuid: "b9235c169d85402faf437d72e9a38bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc"
I20260810 22:05:48.039006 5059 raft_consensus.cc:3037] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:48.040620 4684 raft_consensus.cc:2454] T 778d8c794af64277b2f497708012e16c P 2b27c8ff42ea450bb5114ab97c1ed25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9235c169d85402faf437d72e9a38bdd in term 1.
I20260810 22:05:48.045783 4781 leader_election.cc:302] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd; no voters:
I20260810 22:05:48.046557 5227 consensus_queue.cc:1035] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:48.047657 5188 raft_consensus.cc:2781] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:05:48.048166 5188 raft_consensus.cc:686] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd [term 1 LEADER]: Becoming Leader. State: Replica: b9235c169d85402faf437d72e9a38bdd, State: Running, Role: LEADER
I20260810 22:05:48.048853 5059 raft_consensus.cc:2454] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9235c169d85402faf437d72e9a38bdd in term 1.
I20260810 22:05:48.049567 5188 consensus_queue.cc:237] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:48.062413 4470 catalog_manager.cc:5495] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd reported cstate change: term changed from 0 to 1, leader changed from <none> to b9235c169d85402faf437d72e9a38bdd (127.27.17.130). New cstate: current_term: 1 leader_uuid: "b9235c169d85402faf437d72e9a38bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } health_report { overall_health: UNKNOWN } } }
I20260810 22:05:48.064100 5059 raft_consensus.cc:3037] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:48.074457 5185 consensus_queue.cc:1035] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:48.125653 5185 consensus_queue.cc:1035] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:48.136682 5185 consensus_queue.cc:1035] T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:48.139673 5220 raft_consensus.cc:482] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:48.140079 5220 raft_consensus.cc:504] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:48.142042 5220 leader_election.cc:288] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
I20260810 22:05:48.142472 5220 raft_consensus.cc:482] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:05:48.142841 5220 raft_consensus.cc:504] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:05:48.143330 5059 raft_consensus.cc:3037] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:48.143947 5058 raft_consensus.cc:3037] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:48.144430 5220 leader_election.cc:288] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
I20260810 22:05:48.145845 4874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33e69aaf4a64216b2f995a9e9e9e1e2" candidate_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9235c169d85402faf437d72e9a38bdd" is_pre_election: true
I20260810 22:05:48.146503 4874 raft_consensus.cc:2379] T e33e69aaf4a64216b2f995a9e9e9e1e2 P b9235c169d85402faf437d72e9a38bdd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a552c4b7a11b4cb7a2fdaacbcdf219dc in current term 1: Already voted for candidate 2b27c8ff42ea450bb5114ab97c1ed25d in this term.
I20260810 22:05:48.147248 4872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e619eaa340248da8a54a913c1d16b9f" candidate_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9235c169d85402faf437d72e9a38bdd" is_pre_election: true
I20260810 22:05:48.147910 4872 raft_consensus.cc:2379] T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a552c4b7a11b4cb7a2fdaacbcdf219dc in current term 1: Already voted for candidate 2b27c8ff42ea450bb5114ab97c1ed25d in this term.
I20260810 22:05:48.151024 5246 consensus_queue.cc:1035] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:48.151889 4684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33e69aaf4a64216b2f995a9e9e9e1e2" candidate_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" is_pre_election: true
I20260810 22:05:48.154165 5246 consensus_queue.cc:1035] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:48.154362 4681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e619eaa340248da8a54a913c1d16b9f" candidate_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" is_pre_election: true
I20260810 22:05:48.157091 4971 leader_election.cc:302] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:05:48.158731 4971 leader_election.cc:302] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:05:48.159535 5220 raft_consensus.cc:3037] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:05:48.166391 5220 raft_consensus.cc:2726] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 22:05:48.166360 5216 raft_consensus.cc:2726] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 22:05:48.176347 5246 consensus_queue.cc:1035] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:48.177477 5176 consensus_queue.cc:1035] T 416ba6bb2af24654a64eb1a23261254b P 2b27c8ff42ea450bb5114ab97c1ed25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:48.191938 5176 consensus_queue.cc:1035] T e33e69aaf4a64216b2f995a9e9e9e1e2 P 2b27c8ff42ea450bb5114ab97c1ed25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 22:05:48.246837 5246 consensus_queue.cc:1035] T 8e619eaa340248da8a54a913c1d16b9f P 2b27c8ff42ea450bb5114ab97c1ed25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 22:05:48.310256 5138 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.27.17.190:44453, user_credentials={real_user=slave}} blocked reactor thread for 54441.2us
I20260810 22:05:48.446187 5182 consensus_queue.cc:1035] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:48.457505 5188 consensus_queue.cc:1035] T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:05:48.717772 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling FlushMRSOp(af1fb81326b54dbbb23c7b577bdaf751): perf score=0.001678
W20260810 22:05:48.712438 4591 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.131:44793, user_credentials={real_user=slave}} blocked reactor thread for 93012.1us
W20260810 22:05:48.769541 4595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.130:37933, user_credentials={real_user=slave}} blocked reactor thread for 82057.9us
I20260810 22:05:48.859057 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(e33e69aaf4a64216b2f995a9e9e9e1e2): perf score=0.003577
I20260810 22:05:48.851765 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling FlushMRSOp(416ba6bb2af24654a64eb1a23261254b): perf score=0.000728
W20260810 22:05:49.078902 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.131:44793, user_credentials={real_user=slave}} blocked reactor thread for 180484us
I20260810 22:05:49.413520 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(e33e69aaf4a64216b2f995a9e9e9e1e2) complete. Timing: real 0.537s user 0.064s sys 0.017s Metrics: {"bytes_written":4993,"cfile_init":1,"compiler_manager_pool.queue_time_us":919,"dirs.queue_time_us":20085,"dirs.run_cpu_time_us":334,"dirs.run_wall_time_us":2910,"drs_written":1,"lbm_read_time_us":367,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3098,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":15,"thread_start_us":56159,"threads_started":2}
I20260810 22:05:49.420104 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(416ba6bb2af24654a64eb1a23261254b): perf score=0.003577
W20260810 22:05:49.596180 4595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.130:37933, user_credentials={real_user=slave}} blocked reactor thread for 117098us
W20260810 22:05:49.681211 5288 log.cc:924] Time spent T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
W20260810 22:05:49.873926 4591 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.131:44793, user_credentials={real_user=slave}} blocked reactor thread for 63851.2us
W20260810 22:05:49.878580 4595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.130:37933, user_credentials={real_user=slave}} blocked reactor thread for 61386.1us
I20260810 22:05:50.059010 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: FlushMRSOp(af1fb81326b54dbbb23c7b577bdaf751) complete. Timing: real 1.336s user 0.079s sys 0.081s Metrics: {"bytes_written":4781,"cfile_init":1,"compiler_manager_pool.queue_time_us":102844,"dirs.queue_time_us":265475,"dirs.run_cpu_time_us":376,"dirs.run_wall_time_us":11324,"drs_written":1,"lbm_read_time_us":396,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3786,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":7,"thread_start_us":386215,"threads_started":2}
I20260810 22:05:50.077649 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling UndoDeltaBlockGCOp(af1fb81326b54dbbb23c7b577bdaf751): 192 bytes on disk
W20260810 22:05:50.066555 5348 log.cc:924] Time spent T 518b19decff74b1baf253900da181d46 P b9235c169d85402faf437d72e9a38bdd: Append to log took a long time: real 0.209s user 0.015s sys 0.040s
I20260810 22:05:50.080039 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: UndoDeltaBlockGCOp(af1fb81326b54dbbb23c7b577bdaf751) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":357,"lbm_reads_lt_1ms":4}
W20260810 22:05:50.044200 5244 log.cc:924] Time spent T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P b9235c169d85402faf437d72e9a38bdd: Append to log took a long time: real 0.220s user 0.022s sys 0.035s
I20260810 22:05:50.088114 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.667s user 0.031s sys 0.017s Metrics: {"bytes_written":5156,"cfile_init":1,"dirs.queue_time_us":171171,"dirs.run_cpu_time_us":403,"dirs.run_wall_time_us":7909,"drs_written":1,"lbm_read_time_us":334,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2941,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":20,"thread_start_us":164389,"threads_started":1}
I20260810 22:05:50.089643 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling FlushMRSOp(416ba6bb2af24654a64eb1a23261254b): perf score=0.007377
I20260810 22:05:50.103298 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(416ba6bb2af24654a64eb1a23261254b): 316 bytes on disk
W20260810 22:05:50.052165 5298 log.cc:924] Time spent T 416ba6bb2af24654a64eb1a23261254b P b9235c169d85402faf437d72e9a38bdd: Append to log took a long time: real 0.157s user 0.005s sys 0.006s
W20260810 22:05:50.052258 5295 log.cc:924] Time spent T 8e619eaa340248da8a54a913c1d16b9f P b9235c169d85402faf437d72e9a38bdd: Append to log took a long time: real 0.109s user 0.009s sys 0.002s
W20260810 22:05:50.052047 5276 log.cc:924] Time spent T af1fb81326b54dbbb23c7b577bdaf751 P b9235c169d85402faf437d72e9a38bdd: Append to log took a long time: real 0.136s user 0.003s sys 0.009s
I20260810 22:05:50.125878 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.019s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":431,"lbm_reads_lt_1ms":4}
I20260810 22:05:50.150171 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(e33e69aaf4a64216b2f995a9e9e9e1e2): 282 bytes on disk
I20260810 22:05:50.152072 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(e33e69aaf4a64216b2f995a9e9e9e1e2) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":327,"lbm_reads_lt_1ms":4}
I20260810 22:05:50.161695 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(8e619eaa340248da8a54a913c1d16b9f): perf score=0.007377
I20260810 22:05:50.500155 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: FlushMRSOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.361s user 0.046s sys 0.017s Metrics: {"bytes_written":5844,"cfile_init":1,"dirs.queue_time_us":94178,"dirs.run_cpu_time_us":385,"dirs.run_wall_time_us":23892,"drs_written":1,"lbm_read_time_us":314,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5996,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":46,"thread_start_us":93886,"threads_started":1}
I20260810 22:05:50.565889 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling UndoDeltaBlockGCOp(416ba6bb2af24654a64eb1a23261254b): 568 bytes on disk
I20260810 22:05:50.570220 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: UndoDeltaBlockGCOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":397,"lbm_reads_lt_1ms":4}
W20260810 22:05:50.630460 4591 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.131:44793, user_credentials={real_user=slave}} blocked reactor thread for 115578us
I20260810 22:05:50.640151 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling FlushMRSOp(518b19decff74b1baf253900da181d46): perf score=0.007377
I20260810 22:05:50.608258 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: FlushMRSOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 1.704s user 0.076s sys 0.054s Metrics: {"bytes_written":4711,"cfile_init":1,"compiler_manager_pool.queue_time_us":81131,"dirs.queue_time_us":219204,"dirs.run_cpu_time_us":360,"dirs.run_wall_time_us":8032,"drs_written":1,"lbm_read_time_us":358,"lbm_reads_lt_1ms":4,"lbm_write_time_us":221482,"lbm_writes_1-10_ms":3,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":19,"peak_mem_usage":0,"rows_written":3,"spinlock_wait_cycles":16640,"thread_start_us":303265,"threads_started":2}
I20260810 22:05:50.651319 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling FlushMRSOp(518b19decff74b1baf253900da181d46): perf score=0.007377
I20260810 22:05:50.877712 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: FlushMRSOp(518b19decff74b1baf253900da181d46) complete. Timing: real 0.222s user 0.046s sys 0.004s Metrics: {"bytes_written":5754,"cfile_init":1,"dirs.queue_time_us":414,"dirs.run_cpu_time_us":303,"dirs.run_wall_time_us":1590,"drs_written":1,"lbm_read_time_us":347,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2827,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":41}
I20260810 22:05:50.912458 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling UndoDeltaBlockGCOp(518b19decff74b1baf253900da181d46): 528 bytes on disk
I20260810 22:05:50.939703 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: UndoDeltaBlockGCOp(518b19decff74b1baf253900da181d46) complete. Timing: real 0.008s user 0.002s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":377,"lbm_reads_lt_1ms":4}
I20260810 22:05:50.946327 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling UndoDeltaBlockGCOp(416ba6bb2af24654a64eb1a23261254b): 171 bytes on disk
I20260810 22:05:50.948369 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: UndoDeltaBlockGCOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":382,"lbm_reads_lt_1ms":4}
I20260810 22:05:50.950388 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(8e619eaa340248da8a54a913c1d16b9f) complete. Timing: real 0.781s user 0.046s sys 0.004s Metrics: {"bytes_written":5650,"cfile_init":1,"dirs.queue_time_us":27975,"dirs.run_cpu_time_us":310,"dirs.run_wall_time_us":1047,"drs_written":1,"lbm_read_time_us":262,"lbm_reads_lt_1ms":4,"lbm_write_time_us":416291,"lbm_writes_10-100_ms":2,"lbm_writes_gt_100_ms":2,"lbm_writes_lt_1ms":19,"peak_mem_usage":0,"rows_written":39,"thread_start_us":29184,"threads_started":1}
I20260810 22:05:50.968235 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling FlushMRSOp(628379a2077c4ea7896784b089a5ea0c): perf score=0.007377
I20260810 22:05:50.984270 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(518b19decff74b1baf253900da181d46): perf score=0.007377
W20260810 22:05:51.101758 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.131:44793, user_credentials={real_user=slave}} blocked reactor thread for 140316us
I20260810 22:05:51.134689 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: FlushMRSOp(518b19decff74b1baf253900da181d46) complete. Timing: real 0.494s user 0.040s sys 0.018s Metrics: {"bytes_written":5754,"cfile_init":1,"dirs.queue_time_us":12252,"dirs.run_cpu_time_us":327,"dirs.run_wall_time_us":991,"drs_written":1,"lbm_read_time_us":524,"lbm_reads_lt_1ms":4,"lbm_write_time_us":43710,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":41}
I20260810 22:05:51.152956 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling FlushMRSOp(628379a2077c4ea7896784b089a5ea0c): perf score=0.007377
I20260810 22:05:51.153638 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: FlushMRSOp(628379a2077c4ea7896784b089a5ea0c) complete. Timing: real 0.183s user 0.047s sys 0.005s Metrics: {"bytes_written":5844,"cfile_init":1,"dirs.queue_time_us":9575,"dirs.run_cpu_time_us":324,"dirs.run_wall_time_us":1926,"drs_written":1,"lbm_read_time_us":274,"lbm_reads_lt_1ms":4,"lbm_write_time_us":18520,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":46}
I20260810 22:05:51.159873 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling UndoDeltaBlockGCOp(628379a2077c4ea7896784b089a5ea0c): 570 bytes on disk
I20260810 22:05:51.167043 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: UndoDeltaBlockGCOp(628379a2077c4ea7896784b089a5ea0c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":307,"lbm_reads_lt_1ms":4}
I20260810 22:05:51.198853 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling FlushMRSOp(dc4801ac5aeb4b4d8340ee2f5c0d73f6): perf score=0.014976
W20260810 22:05:51.344141 5297 meta_cache.cc:1261] Time spent looking up entry by key: real 0.063s user 0.003s sys 0.000s
W20260810 22:05:51.484282 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.129:43093, user_credentials={real_user=slave}} blocked reactor thread for 49855.9us
I20260810 22:05:51.496155 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(518b19decff74b1baf253900da181d46) complete. Timing: real 0.486s user 0.042s sys 0.008s Metrics: {"bytes_written":5754,"cfile_init":1,"dirs.queue_time_us":82583,"dirs.run_cpu_time_us":333,"dirs.run_wall_time_us":3832,"drs_written":1,"lbm_read_time_us":325,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12246,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":20,"peak_mem_usage":0,"rows_written":41,"thread_start_us":18651,"threads_started":1}
I20260810 22:05:51.559490 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(518b19decff74b1baf253900da181d46): 528 bytes on disk
I20260810 22:05:51.565495 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: FlushMRSOp(628379a2077c4ea7896784b089a5ea0c) complete. Timing: real 0.407s user 0.032s sys 0.019s Metrics: {"bytes_written":5844,"cfile_init":1,"dirs.queue_time_us":5147,"dirs.run_cpu_time_us":284,"dirs.run_wall_time_us":1578,"drs_written":1,"lbm_read_time_us":11418,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":2553,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":46,"thread_start_us":34711,"threads_started":1}
I20260810 22:05:51.561256 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(518b19decff74b1baf253900da181d46) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":285,"lbm_reads_lt_1ms":4}
I20260810 22:05:51.573966 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling UndoDeltaBlockGCOp(628379a2077c4ea7896784b089a5ea0c): 570 bytes on disk
I20260810 22:05:51.576517 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: UndoDeltaBlockGCOp(628379a2077c4ea7896784b089a5ea0c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":273,"lbm_reads_lt_1ms":4}
I20260810 22:05:51.581413 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(8e619eaa340248da8a54a913c1d16b9f): 495 bytes on disk
I20260810 22:05:51.586971 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling FlushMRSOp(8e619eaa340248da8a54a913c1d16b9f): perf score=0.014976
I20260810 22:05:51.603606 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(8e619eaa340248da8a54a913c1d16b9f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":305,"lbm_reads_lt_1ms":4}
I20260810 22:05:51.627714 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(af1fb81326b54dbbb23c7b577bdaf751): perf score=0.014976
I20260810 22:05:51.756485 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: FlushMRSOp(8e619eaa340248da8a54a913c1d16b9f) complete. Timing: real 0.168s user 0.040s sys 0.012s Metrics: {"bytes_written":6715,"cfile_init":1,"dirs.queue_time_us":9512,"dirs.run_cpu_time_us":330,"dirs.run_wall_time_us":9710,"drs_written":1,"lbm_read_time_us":250,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2421,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":79,"thread_start_us":19329,"threads_started":1}
I20260810 22:05:51.772596 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: FlushMRSOp(dc4801ac5aeb4b4d8340ee2f5c0d73f6) complete. Timing: real 0.573s user 0.049s sys 0.010s Metrics: {"bytes_written":6466,"cfile_init":1,"dirs.queue_time_us":58077,"dirs.run_cpu_time_us":336,"dirs.run_wall_time_us":36971,"drs_written":1,"lbm_read_time_us":285,"lbm_reads_lt_1ms":4,"lbm_write_time_us":36132,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":69,"thread_start_us":57768,"threads_started":1}
I20260810 22:05:51.783968 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling FlushMRSOp(af1fb81326b54dbbb23c7b577bdaf751): perf score=0.014976
I20260810 22:05:51.798892 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling UndoDeltaBlockGCOp(dc4801ac5aeb4b4d8340ee2f5c0d73f6): 785 bytes on disk
I20260810 22:05:51.800652 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: UndoDeltaBlockGCOp(dc4801ac5aeb4b4d8340ee2f5c0d73f6) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":283,"lbm_reads_lt_1ms":4}
I20260810 22:05:51.806250 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling FlushMRSOp(416ba6bb2af24654a64eb1a23261254b): perf score=0.014976
W20260810 22:05:51.835487 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.131:44793, user_credentials={real_user=slave}} blocked reactor thread for 45000.2us
I20260810 22:05:52.109606 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: FlushMRSOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.303s user 0.064s sys 0.006s Metrics: {"bytes_written":6908,"cfile_init":1,"dirs.queue_time_us":58372,"dirs.run_cpu_time_us":347,"dirs.run_wall_time_us":5809,"drs_written":1,"lbm_read_time_us":302,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2557,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":86,"thread_start_us":58634,"threads_started":1}
I20260810 22:05:52.134352 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling UndoDeltaBlockGCOp(416ba6bb2af24654a64eb1a23261254b): 954 bytes on disk
I20260810 22:05:52.136233 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: UndoDeltaBlockGCOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":280,"lbm_reads_lt_1ms":4}
I20260810 22:05:52.154572 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling CompactRowSetsOp(416ba6bb2af24654a64eb1a23261254b): perf score=0.652897
W20260810 22:05:52.156598 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.131:44793, user_credentials={real_user=slave}} blocked reactor thread for 92172.7us
I20260810 22:05:52.441397 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(af1fb81326b54dbbb23c7b577bdaf751) complete. Timing: real 0.726s user 0.077s sys 0.011s Metrics: {"bytes_written":6908,"cfile_init":1,"dirs.queue_time_us":158343,"dirs.run_cpu_time_us":317,"dirs.run_wall_time_us":13094,"drs_written":1,"lbm_read_time_us":266,"lbm_reads_lt_1ms":4,"lbm_write_time_us":24760,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":86,"spinlock_wait_cycles":648064,"thread_start_us":156138,"threads_started":1}
I20260810 22:05:52.466374 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(af1fb81326b54dbbb23c7b577bdaf751): 952 bytes on disk
I20260810 22:05:52.468056 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(af1fb81326b54dbbb23c7b577bdaf751) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":329,"lbm_reads_lt_1ms":4}
I20260810 22:05:52.474910 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(416ba6bb2af24654a64eb1a23261254b): perf score=0.014976
I20260810 22:05:52.554097 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: CompactRowSetsOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.399s user 0.064s sys 0.008s Metrics: {"bytes_written":7023,"cfile_cache_hit":10,"cfile_cache_hit_bytes":1848,"cfile_cache_miss":26,"cfile_cache_miss_bytes":5634,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":37338,"dirs.run_cpu_time_us":1865,"dirs.run_wall_time_us":23888,"drs_written":1,"lbm_read_time_us":3151,"lbm_reads_lt_1ms":46,"lbm_write_time_us":2358,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":1131,"rows_written":89,"spinlock_wait_cycles":7936,"thread_start_us":39274,"threads_started":6}
I20260810 22:05:52.561070 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling FlushMRSOp(778d8c794af64277b2f497708012e16c): perf score=0.014976
I20260810 22:05:52.585242 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.110s user 0.048s sys 0.001s Metrics: {"bytes_written":7039,"cfile_init":1,"dirs.queue_time_us":3886,"dirs.run_cpu_time_us":288,"dirs.run_wall_time_us":1385,"drs_written":1,"lbm_read_time_us":273,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2275,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":90}
I20260810 22:05:52.600791 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(416ba6bb2af24654a64eb1a23261254b): 994 bytes on disk
I20260810 22:05:52.606307 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":276,"lbm_reads_lt_1ms":4}
I20260810 22:05:52.616209 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling CompactRowSetsOp(416ba6bb2af24654a64eb1a23261254b): perf score=0.946002
W20260810 22:05:52.685308 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.129:43093, user_credentials={real_user=slave}} blocked reactor thread for 87611.6us
I20260810 22:05:52.715242 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: FlushMRSOp(af1fb81326b54dbbb23c7b577bdaf751) complete. Timing: real 0.930s user 0.047s sys 0.013s Metrics: {"bytes_written":6715,"cfile_init":1,"dirs.queue_time_us":91,"dirs.run_cpu_time_us":320,"dirs.run_wall_time_us":972,"drs_written":1,"lbm_read_time_us":256,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2455,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":79}
I20260810 22:05:52.750721 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling CompactRowSetsOp(af1fb81326b54dbbb23c7b577bdaf751): perf score=0.897718
W20260810 22:05:52.854952 5291 meta_cache.cc:1261] Time spent looking up entry by key: real 0.053s user 0.004s sys 0.000s
I20260810 22:05:53.019737 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: FlushMRSOp(778d8c794af64277b2f497708012e16c) complete. Timing: real 0.458s user 0.056s sys 0.008s Metrics: {"bytes_written":7246,"cfile_init":1,"dirs.queue_time_us":80,"dirs.run_cpu_time_us":295,"dirs.run_wall_time_us":1981,"drs_written":1,"lbm_read_time_us":266,"lbm_reads_lt_1ms":4,"lbm_write_time_us":64084,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":20,"peak_mem_usage":0,"rows_written":98}
I20260810 22:05:53.041074 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling UndoDeltaBlockGCOp(778d8c794af64277b2f497708012e16c): 1070 bytes on disk
I20260810 22:05:53.150784 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: CompactRowSetsOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.525s user 0.089s sys 0.035s Metrics: {"bytes_written":7541,"cfile_cache_hit":10,"cfile_cache_hit_bytes":2218,"cfile_cache_miss":26,"cfile_cache_miss_bytes":6776,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":86012,"dirs.run_cpu_time_us":2953,"dirs.run_wall_time_us":30632,"drs_written":1,"lbm_read_time_us":12037,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":45,"lbm_write_time_us":2412,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":1184,"rows_written":110,"thread_start_us":285081,"threads_started":5}
I20260810 22:05:53.160221 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(416ba6bb2af24654a64eb1a23261254b): 1193 bytes on disk
I20260810 22:05:53.162359 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":263,"lbm_reads_lt_1ms":4}
I20260810 22:05:53.169777 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(628379a2077c4ea7896784b089a5ea0c): perf score=0.014976
I20260810 22:05:53.315821 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: UndoDeltaBlockGCOp(778d8c794af64277b2f497708012e16c) complete. Timing: real 0.003s user 0.000s sys 0.001s Metrics: {"cfile_init":1,"lbm_read_time_us":661,"lbm_reads_lt_1ms":4}
W20260810 22:05:53.330127 5254 log.cc:924] Time spent T f0037195580e420fad2a77957025fc74 P b9235c169d85402faf437d72e9a38bdd: Append to log took a long time: real 0.166s user 0.000s sys 0.000s
W20260810 22:05:53.333123 5283 log.cc:924] Time spent T 778d8c794af64277b2f497708012e16c P b9235c169d85402faf437d72e9a38bdd: Append to log took a long time: real 0.165s user 0.000s sys 0.000s
I20260810 22:05:53.352257 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling FlushMRSOp(8e619eaa340248da8a54a913c1d16b9f): perf score=0.014976
I20260810 22:05:53.422094 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: CompactRowSetsOp(af1fb81326b54dbbb23c7b577bdaf751) complete. Timing: real 0.670s user 0.100s sys 0.097s Metrics: {"bytes_written":6908,"cfile_cache_hit":10,"cfile_cache_hit_bytes":1788,"cfile_cache_miss":26,"cfile_cache_miss_bytes":5346,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":78145,"dirs.run_cpu_time_us":2593,"dirs.run_wall_time_us":165524,"drs_written":1,"lbm_read_time_us":3149,"lbm_reads_lt_1ms":50,"lbm_write_time_us":4566,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"peak_mem_usage":1037,"rows_written":86,"thread_start_us":194339,"threads_started":5}
I20260810 22:05:53.430472 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling FlushMRSOp(416ba6bb2af24654a64eb1a23261254b): perf score=0.014976
I20260810 22:05:53.685514 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(628379a2077c4ea7896784b089a5ea0c) complete. Timing: real 0.514s user 0.058s sys 0.004s Metrics: {"bytes_written":6715,"cfile_init":1,"dirs.queue_time_us":180,"dirs.run_cpu_time_us":319,"dirs.run_wall_time_us":927,"drs_written":1,"lbm_read_time_us":278,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2652,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":79}
I20260810 22:05:53.709957 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(778d8c794af64277b2f497708012e16c): perf score=0.014976
W20260810 22:05:53.863735 5357 log.cc:924] Time spent T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P 2b27c8ff42ea450bb5114ab97c1ed25d: Append to log took a long time: real 0.067s user 0.000s sys 0.008s
I20260810 22:05:53.956421 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(778d8c794af64277b2f497708012e16c) complete. Timing: real 0.246s user 0.059s sys 0.018s Metrics: {"bytes_written":7886,"cfile_init":1,"dirs.queue_time_us":84,"dirs.run_cpu_time_us":301,"dirs.run_wall_time_us":867,"drs_written":1,"lbm_read_time_us":314,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2679,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":122}
I20260810 22:05:53.963935 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(778d8c794af64277b2f497708012e16c): 1300 bytes on disk
I20260810 22:05:53.966547 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(778d8c794af64277b2f497708012e16c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":299,"lbm_reads_lt_1ms":4}
I20260810 22:05:54.031708 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(518b19decff74b1baf253900da181d46): perf score=0.014976
I20260810 22:05:54.126408 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: FlushMRSOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.695s user 0.052s sys 0.000s Metrics: {"bytes_written":6500,"cfile_init":1,"dirs.queue_time_us":81037,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":1428,"drs_written":1,"lbm_read_time_us":287,"lbm_reads_lt_1ms":4,"lbm_write_time_us":102675,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":71}
I20260810 22:05:54.138244 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling CompactRowSetsOp(416ba6bb2af24654a64eb1a23261254b): perf score=1.007041
W20260810 22:05:54.149843 5299 meta_cache.cc:1261] Time spent looking up entry by key: real 0.081s user 0.003s sys 0.000s
I20260810 22:05:54.507467 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: FlushMRSOp(8e619eaa340248da8a54a913c1d16b9f) complete. Timing: real 1.154s user 0.066s sys 0.020s Metrics: {"bytes_written":7699,"cfile_init":1,"dirs.queue_time_us":94898,"dirs.run_cpu_time_us":335,"dirs.run_wall_time_us":9427,"drs_written":1,"lbm_read_time_us":317,"lbm_reads_lt_1ms":4,"lbm_write_time_us":74826,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":115,"thread_start_us":120342,"threads_started":1}
I20260810 22:05:54.604128 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling UndoDeltaBlockGCOp(8e619eaa340248da8a54a913c1d16b9f): 1230 bytes on disk
I20260810 22:05:54.606747 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: UndoDeltaBlockGCOp(8e619eaa340248da8a54a913c1d16b9f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":265,"lbm_reads_lt_1ms":4}
I20260810 22:05:54.614928 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling FlushMRSOp(af1fb81326b54dbbb23c7b577bdaf751): perf score=0.030173
I20260810 22:05:54.697263 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(518b19decff74b1baf253900da181d46) complete. Timing: real 0.662s user 0.054s sys 0.000s Metrics: {"bytes_written":6642,"cfile_init":1,"dirs.queue_time_us":67316,"dirs.run_cpu_time_us":286,"dirs.run_wall_time_us":55076,"drs_written":1,"lbm_read_time_us":282,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2461,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":75,"thread_start_us":81389,"threads_started":1}
I20260810 22:05:54.705986 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling CompactRowSetsOp(518b19decff74b1baf253900da181d46): perf score=1.000577
I20260810 22:05:54.814014 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: CompactRowSetsOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.675s user 0.086s sys 0.030s Metrics: {"bytes_written":7733,"cfile_cache_hit":10,"cfile_cache_hit_bytes":2349,"cfile_cache_miss":26,"cfile_cache_miss_bytes":7067,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":100832,"dirs.run_cpu_time_us":2115,"dirs.run_wall_time_us":12328,"drs_written":1,"lbm_read_time_us":35647,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":2586,"lbm_writes_lt_1ms":23,"mutex_wait_us":579,"num_input_rowsets":2,"peak_mem_usage":601,"rows_written":117,"thread_start_us":287529,"threads_started":5}
I20260810 22:05:54.824231 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling FlushMRSOp(dc4801ac5aeb4b4d8340ee2f5c0d73f6): perf score=0.030173
W20260810 22:05:55.259044 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.129:43093, user_credentials={real_user=slave}} blocked reactor thread for 70219.1us
I20260810 22:05:55.288395 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: FlushMRSOp(af1fb81326b54dbbb23c7b577bdaf751) complete. Timing: real 0.655s user 0.086s sys 0.019s Metrics: {"bytes_written":8813,"cfile_init":1,"dirs.queue_time_us":218919,"dirs.run_cpu_time_us":325,"dirs.run_wall_time_us":10445,"drs_written":1,"lbm_read_time_us":284,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2903,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":159,"thread_start_us":222988,"threads_started":1}
I20260810 22:05:55.302834 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling UndoDeltaBlockGCOp(af1fb81326b54dbbb23c7b577bdaf751): 1659 bytes on disk
W20260810 22:05:55.310122 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.131:44793, user_credentials={real_user=slave}} blocked reactor thread for 49412.5us
I20260810 22:05:55.310948 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: UndoDeltaBlockGCOp(af1fb81326b54dbbb23c7b577bdaf751) complete. Timing: real 0.004s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":343,"lbm_reads_lt_1ms":4}
I20260810 22:05:55.389233 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: FlushMRSOp(dc4801ac5aeb4b4d8340ee2f5c0d73f6) complete. Timing: real 0.564s user 0.076s sys 0.000s Metrics: {"bytes_written":8587,"cfile_init":1,"dirs.queue_time_us":7107,"dirs.run_cpu_time_us":372,"dirs.run_wall_time_us":5660,"drs_written":1,"lbm_read_time_us":325,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6888,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":150}
I20260810 22:05:55.451874 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling FlushMRSOp(778d8c794af64277b2f497708012e16c): perf score=0.030173
W20260810 22:05:55.589425 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.129:43093, user_credentials={real_user=slave}} blocked reactor thread for 117956us
I20260810 22:05:55.729465 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling FlushMRSOp(e33e69aaf4a64216b2f995a9e9e9e1e2): perf score=0.030173
I20260810 22:05:55.805692 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: FlushMRSOp(778d8c794af64277b2f497708012e16c) complete. Timing: real 0.346s user 0.079s sys 0.004s Metrics: {"bytes_written":8587,"cfile_init":1,"dirs.queue_time_us":4041,"dirs.run_cpu_time_us":370,"dirs.run_wall_time_us":40765,"drs_written":1,"lbm_read_time_us":400,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6209,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":150,"thread_start_us":20733,"threads_started":1}
I20260810 22:05:55.829046 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling FlushMRSOp(e33e69aaf4a64216b2f995a9e9e9e1e2): perf score=0.030173
W20260810 22:05:55.922557 5300 meta_cache.cc:1261] Time spent looking up entry by key: real 0.057s user 0.004s sys 0.000s
W20260810 22:05:55.988951 5297 meta_cache.cc:1261] Time spent looking up entry by key: real 0.050s user 0.000s sys 0.000s
I20260810 22:05:56.441936 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: CompactRowSetsOp(518b19decff74b1baf253900da181d46) complete. Timing: real 1.735s user 0.110s sys 0.042s Metrics: {"bytes_written":7716,"cfile_cache_hit":10,"cfile_cache_hit_bytes":2332,"cfile_cache_miss":26,"cfile_cache_miss_bytes":7206,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":286964,"dirs.run_cpu_time_us":2990,"dirs.run_wall_time_us":77881,"drs_written":1,"lbm_read_time_us":4832,"lbm_reads_lt_1ms":50,"lbm_write_time_us":69235,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":21,"mutex_wait_us":3119,"num_input_rowsets":2,"peak_mem_usage":984,"rows_written":116,"thread_start_us":355189,"threads_started":6}
I20260810 22:05:56.466348 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(dc4801ac5aeb4b4d8340ee2f5c0d73f6): perf score=0.030173
I20260810 22:05:56.483875 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: FlushMRSOp(e33e69aaf4a64216b2f995a9e9e9e1e2) complete. Timing: real 0.753s user 0.081s sys 0.007s Metrics: {"bytes_written":9029,"cfile_init":1,"dirs.queue_time_us":107867,"dirs.run_cpu_time_us":343,"dirs.run_wall_time_us":13989,"drs_written":1,"lbm_read_time_us":312,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6213,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":167,"thread_start_us":73381,"threads_started":1}
W20260810 22:05:56.407673 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.129:43093, user_credentials={real_user=slave}} blocked reactor thread for 92440.3us
I20260810 22:05:56.500231 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling UndoDeltaBlockGCOp(e33e69aaf4a64216b2f995a9e9e9e1e2): 1764 bytes on disk
I20260810 22:05:56.506652 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: UndoDeltaBlockGCOp(e33e69aaf4a64216b2f995a9e9e9e1e2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":330,"lbm_reads_lt_1ms":4}
I20260810 22:05:56.526381 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling UndoDeltaBlockGCOp(416ba6bb2af24654a64eb1a23261254b): 986 bytes on disk
I20260810 22:05:56.528354 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: UndoDeltaBlockGCOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":329,"lbm_reads_lt_1ms":4}
I20260810 22:05:56.530581 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: FlushMRSOp(e33e69aaf4a64216b2f995a9e9e9e1e2) complete. Timing: real 0.700s user 0.066s sys 0.010s Metrics: {"bytes_written":9132,"cfile_init":1,"dirs.queue_time_us":10508,"dirs.run_cpu_time_us":283,"dirs.run_wall_time_us":6997,"drs_written":1,"lbm_read_time_us":247,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3749,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":169,"thread_start_us":34481,"threads_started":1}
I20260810 22:05:56.538396 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling UndoDeltaBlockGCOp(e33e69aaf4a64216b2f995a9e9e9e1e2): 1788 bytes on disk
I20260810 22:05:56.554005 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling FlushMRSOp(518b19decff74b1baf253900da181d46): perf score=0.030173
I20260810 22:05:56.556975 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: UndoDeltaBlockGCOp(e33e69aaf4a64216b2f995a9e9e9e1e2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":286,"lbm_reads_lt_1ms":4}
I20260810 22:05:56.578563 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling UndoDeltaBlockGCOp(dc4801ac5aeb4b4d8340ee2f5c0d73f6): 1595 bytes on disk
I20260810 22:05:56.588860 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: UndoDeltaBlockGCOp(dc4801ac5aeb4b4d8340ee2f5c0d73f6) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":301,"lbm_reads_lt_1ms":4}
I20260810 22:05:56.608253 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling UndoDeltaBlockGCOp(778d8c794af64277b2f497708012e16c): 1585 bytes on disk
I20260810 22:05:56.619916 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: UndoDeltaBlockGCOp(778d8c794af64277b2f497708012e16c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":266,"lbm_reads_lt_1ms":4}
I20260810 22:05:56.638437 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling FlushMRSOp(518b19decff74b1baf253900da181d46): perf score=0.030173
W20260810 22:05:56.842029 5300 meta_cache.cc:1261] Time spent looking up entry by key: real 0.123s user 0.001s sys 0.000s
W20260810 22:05:57.040199 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.129:43093, user_credentials={real_user=slave}} blocked reactor thread for 188544us
I20260810 22:05:57.116266 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(dc4801ac5aeb4b4d8340ee2f5c0d73f6) complete. Timing: real 0.649s user 0.097s sys 0.010s Metrics: {"bytes_written":9658,"cfile_init":1,"dirs.queue_time_us":77627,"dirs.run_cpu_time_us":401,"dirs.run_wall_time_us":7043,"drs_written":1,"lbm_read_time_us":257,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2804,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":191,"thread_start_us":80730,"threads_started":1}
I20260810 22:05:57.196931 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(dc4801ac5aeb4b4d8340ee2f5c0d73f6): 1994 bytes on disk
I20260810 22:05:57.240513 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(dc4801ac5aeb4b4d8340ee2f5c0d73f6) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":428,"lbm_reads_lt_1ms":4}
I20260810 22:05:57.383559 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(8e619eaa340248da8a54a913c1d16b9f): perf score=0.030173
W20260810 22:05:57.479517 5297 meta_cache.cc:1261] Time spent looking up entry by key: real 0.090s user 0.004s sys 0.000s
I20260810 22:05:57.517894 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: FlushMRSOp(518b19decff74b1baf253900da181d46) complete. Timing: real 0.860s user 0.080s sys 0.003s Metrics: {"bytes_written":8407,"cfile_init":1,"dirs.queue_time_us":24165,"dirs.run_cpu_time_us":416,"dirs.run_wall_time_us":1697,"drs_written":1,"lbm_read_time_us":334,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3014,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":144,"thread_start_us":13400,"threads_started":1}
I20260810 22:05:57.528966 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling UndoDeltaBlockGCOp(518b19decff74b1baf253900da181d46): 2056 bytes on disk
I20260810 22:05:57.532406 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: UndoDeltaBlockGCOp(518b19decff74b1baf253900da181d46) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":521,"lbm_reads_lt_1ms":8}
I20260810 22:05:57.569530 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling CompactRowSetsOp(518b19decff74b1baf253900da181d46): perf score=0.996551
W20260810 22:05:57.732756 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.131:44793, user_credentials={real_user=slave}} blocked reactor thread for 44855.6us
I20260810 22:05:57.785838 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: FlushMRSOp(518b19decff74b1baf253900da181d46) complete. Timing: real 1.220s user 0.070s sys 0.010s Metrics: {"bytes_written":8407,"cfile_init":1,"dirs.queue_time_us":82984,"dirs.run_cpu_time_us":616,"dirs.run_wall_time_us":6388,"drs_written":1,"lbm_read_time_us":375,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6486,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":144,"thread_start_us":83580,"threads_started":1}
I20260810 22:05:57.794049 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling CompactRowSetsOp(518b19decff74b1baf253900da181d46): perf score=0.996551
W20260810 22:05:57.870016 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.131:44793, user_credentials={real_user=slave}} blocked reactor thread for 46491.8us
I20260810 22:05:58.109407 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(8e619eaa340248da8a54a913c1d16b9f) complete. Timing: real 0.664s user 0.119s sys 0.042s Metrics: {"bytes_written":9217,"cfile_init":1,"dirs.queue_time_us":207260,"dirs.run_cpu_time_us":351,"dirs.run_wall_time_us":8273,"drs_written":1,"lbm_read_time_us":308,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6751,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":174,"thread_start_us":206954,"threads_started":1}
I20260810 22:05:58.116490 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling CompactRowSetsOp(8e619eaa340248da8a54a913c1d16b9f): perf score=0.986897
I20260810 22:05:58.455581 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: CompactRowSetsOp(8e619eaa340248da8a54a913c1d16b9f) complete. Timing: real 0.337s user 0.097s sys 0.022s Metrics: {"bytes_written":10250,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4101,"cfile_cache_miss":26,"cfile_cache_miss_bytes":12253,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":79574,"dirs.run_cpu_time_us":3154,"dirs.run_wall_time_us":66918,"drs_written":1,"lbm_read_time_us":14689,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":2209,"lbm_writes_lt_1ms":23,"mutex_wait_us":693,"num_input_rowsets":2,"peak_mem_usage":2804,"rows_written":213,"spinlock_wait_cycles":542080,"thread_start_us":103755,"threads_started":4}
I20260810 22:05:58.486989 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(8e619eaa340248da8a54a913c1d16b9f): 2204 bytes on disk
I20260810 22:05:58.488514 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(8e619eaa340248da8a54a913c1d16b9f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":246,"lbm_reads_lt_1ms":4}
I20260810 22:05:58.519980 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(518b19decff74b1baf253900da181d46): 1262 bytes on disk
I20260810 22:05:58.543884 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(518b19decff74b1baf253900da181d46) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":260,"lbm_reads_lt_1ms":4}
I20260810 22:05:58.570513 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(af1fb81326b54dbbb23c7b577bdaf751): perf score=0.030173
W20260810 22:05:58.658520 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.131:44793, user_credentials={real_user=slave}} blocked reactor thread for 80598.3us
I20260810 22:05:58.813699 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(af1fb81326b54dbbb23c7b577bdaf751) complete. Timing: real 0.236s user 0.084s sys 0.001s Metrics: {"bytes_written":9009,"cfile_init":1,"dirs.queue_time_us":1179,"dirs.run_cpu_time_us":251,"dirs.run_wall_time_us":1505,"drs_written":1,"lbm_read_time_us":311,"lbm_reads_lt_1ms":4,"lbm_write_time_us":15021,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":166}
I20260810 22:05:58.835516 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(af1fb81326b54dbbb23c7b577bdaf751): 1737 bytes on disk
I20260810 22:05:58.837383 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(af1fb81326b54dbbb23c7b577bdaf751) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":304,"lbm_reads_lt_1ms":4}
I20260810 22:05:58.850222 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(628379a2077c4ea7896784b089a5ea0c): 900 bytes on disk
I20260810 22:05:58.858644 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(628379a2077c4ea7896784b089a5ea0c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":300,"lbm_reads_lt_1ms":4}
I20260810 22:05:58.864043 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling CompactRowSetsOp(af1fb81326b54dbbb23c7b577bdaf751): perf score=1.003411
I20260810 22:05:58.914999 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: CompactRowSetsOp(518b19decff74b1baf253900da181d46) complete. Timing: real 1.344s user 0.128s sys 0.075s Metrics: {"bytes_written":9549,"cfile_cache_hit":10,"cfile_cache_hit_bytes":3590,"cfile_cache_miss":26,"cfile_cache_miss_bytes":10816,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":768272,"dirs.run_cpu_time_us":4000,"dirs.run_wall_time_us":159999,"drs_written":1,"lbm_read_time_us":42817,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":45,"lbm_write_time_us":17061,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"peak_mem_usage":2433,"rows_written":185,"thread_start_us":934705,"threads_started":6}
I20260810 22:05:58.926219 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling FlushMRSOp(628379a2077c4ea7896784b089a5ea0c): perf score=0.030173
W20260810 22:05:58.963958 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.131:44793, user_credentials={real_user=slave}} blocked reactor thread for 53002.2us
W20260810 22:05:59.380772 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.129:43093, user_credentials={real_user=slave}} blocked reactor thread for 78840.5us
I20260810 22:05:59.649706 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: CompactRowSetsOp(af1fb81326b54dbbb23c7b577bdaf751) complete. Timing: real 0.763s user 0.112s sys 0.093s Metrics: {"bytes_written":11278,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4814,"cfile_cache_miss":26,"cfile_cache_miss_bytes":14368,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":302298,"dirs.run_cpu_time_us":2480,"dirs.run_wall_time_us":72542,"drs_written":1,"lbm_read_time_us":66265,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":44,"lbm_write_time_us":2770,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":3322,"rows_written":252,"thread_start_us":368194,"threads_started":6}
I20260810 22:05:59.659706 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(518b19decff74b1baf253900da181d46): perf score=0.030173
I20260810 22:05:59.659723 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: CompactRowSetsOp(518b19decff74b1baf253900da181d46) complete. Timing: real 1.843s user 0.103s sys 0.013s Metrics: {"bytes_written":9549,"cfile_cache_hit":10,"cfile_cache_hit_bytes":3590,"cfile_cache_miss":26,"cfile_cache_miss_bytes":10816,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":129056,"dirs.run_cpu_time_us":2938,"dirs.run_wall_time_us":68432,"drs_written":1,"lbm_read_time_us":4025,"lbm_reads_lt_1ms":50,"lbm_write_time_us":43971,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":21,"mutex_wait_us":40588,"num_input_rowsets":2,"peak_mem_usage":2433,"rows_written":185,"thread_start_us":353106,"threads_started":6}
I20260810 22:05:59.667006 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling FlushMRSOp(416ba6bb2af24654a64eb1a23261254b): perf score=0.030173
W20260810 22:05:59.856590 4591 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.131:44793, user_credentials={real_user=slave}} blocked reactor thread for 51795.4us
I20260810 22:05:59.870844 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(518b19decff74b1baf253900da181d46) complete. Timing: real 0.210s user 0.065s sys 0.013s Metrics: {"bytes_written":8324,"cfile_init":1,"dirs.queue_time_us":83,"dirs.run_cpu_time_us":291,"dirs.run_wall_time_us":2269,"drs_written":1,"lbm_read_time_us":236,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2389,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":139}
I20260810 22:05:59.876574 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling CompactRowSetsOp(518b19decff74b1baf253900da181d46): perf score=1.012587
W20260810 22:06:00.015273 4595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.130:37933, user_credentials={real_user=slave}} blocked reactor thread for 49464.1us
I20260810 22:06:00.015045 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: CompactRowSetsOp(518b19decff74b1baf253900da181d46) complete. Timing: real 0.136s user 0.090s sys 0.001s Metrics: {"bytes_written":11335,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4865,"cfile_cache_miss":26,"cfile_cache_miss_bytes":14643,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":759,"dirs.run_cpu_time_us":2042,"dirs.run_wall_time_us":10224,"drs_written":1,"lbm_read_time_us":3531,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2542,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":3358,"rows_written":255}
I20260810 22:06:00.029592 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(416ba6bb2af24654a64eb1a23261254b): perf score=0.030173
I20260810 22:06:00.129904 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: FlushMRSOp(628379a2077c4ea7896784b089a5ea0c) complete. Timing: real 1.195s user 0.079s sys 0.012s Metrics: {"bytes_written":9620,"cfile_init":1,"dirs.queue_time_us":243588,"dirs.run_cpu_time_us":260,"dirs.run_wall_time_us":4041,"drs_written":1,"lbm_read_time_us":262,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2745,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":189,"thread_start_us":243775,"threads_started":1}
I20260810 22:06:00.135965 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling CompactRowSetsOp(628379a2077c4ea7896784b089a5ea0c): perf score=1.012430
W20260810 22:06:00.153157 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.129:43093, user_credentials={real_user=slave}} blocked reactor thread for 70146.4us
I20260810 22:06:00.214918 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.185s user 0.066s sys 0.003s Metrics: {"bytes_written":8704,"cfile_init":1,"dirs.queue_time_us":82,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":1436,"drs_written":1,"lbm_read_time_us":275,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3480,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":153}
I20260810 22:06:00.221516 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(518b19decff74b1baf253900da181d46): 2623 bytes on disk
W20260810 22:06:00.231642 4784 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-2/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-2/data (487964672 bytes available vs 518351011 bytes reserved) (error 28)
I20260810 22:06:00.237509 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(518b19decff74b1baf253900da181d46) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":288,"lbm_reads_lt_1ms":4}
I20260810 22:06:00.245498 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(af1fb81326b54dbbb23c7b577bdaf751): 2575 bytes on disk
I20260810 22:06:00.250650 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(af1fb81326b54dbbb23c7b577bdaf751) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":271,"lbm_reads_lt_1ms":4}
I20260810 22:06:00.257193 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling CompactRowSetsOp(416ba6bb2af24654a64eb1a23261254b): perf score=1.006892
W20260810 22:06:00.313001 4975 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-2/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-2/data (454324224 bytes available vs 518351011 bytes reserved) (error 28)
W20260810 22:06:00.480167 4975 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-2/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-2/data (420712448 bytes available vs 518351011 bytes reserved) (error 28)
I20260810 22:06:00.566867 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: CompactRowSetsOp(628379a2077c4ea7896784b089a5ea0c) complete. Timing: real 0.426s user 0.102s sys 0.020s Metrics: {"bytes_written":10830,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4505,"cfile_cache_miss":26,"cfile_cache_miss_bytes":13465,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":20738,"dirs.run_cpu_time_us":2459,"dirs.run_wall_time_us":17353,"drs_written":1,"lbm_read_time_us":33736,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":47,"lbm_write_time_us":2631,"lbm_writes_lt_1ms":23,"mutex_wait_us":31,"num_input_rowsets":2,"peak_mem_usage":3098,"rows_written":235,"spinlock_wait_cycles":43648,"thread_start_us":145678,"threads_started":4}
I20260810 22:06:00.574450 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling UndoDeltaBlockGCOp(628379a2077c4ea7896784b089a5ea0c): 2426 bytes on disk
I20260810 22:06:00.576144 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: UndoDeltaBlockGCOp(628379a2077c4ea7896784b089a5ea0c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":276,"lbm_reads_lt_1ms":4,"spinlock_wait_cycles":34176}
I20260810 22:06:00.581372 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling FlushMRSOp(f0037195580e420fad2a77957025fc74): perf score=0.060569
I20260810 22:06:00.585480 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: FlushMRSOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.914s user 0.110s sys 0.008s Metrics: {"bytes_written":8939,"cfile_init":1,"dirs.queue_time_us":47565,"dirs.run_cpu_time_us":319,"dirs.run_wall_time_us":3849,"drs_written":1,"lbm_read_time_us":321,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9454,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":162,"spinlock_wait_cycles":4204288,"thread_start_us":47863,"threads_started":1}
I20260810 22:06:00.594717 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling CompactRowSetsOp(416ba6bb2af24654a64eb1a23261254b): perf score=1.006442
W20260810 22:06:00.623265 4596 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1/data (353607680 bytes available vs 518351011 bytes reserved) (error 28)
W20260810 22:06:00.652729 4784 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0/data (320045056 bytes available vs 518351011 bytes reserved) (error 28)
W20260810 22:06:00.684463 4596 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-1/data (286482432 bytes available vs 518351011 bytes reserved) (error 28)
W20260810 22:06:00.706805 4784 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-0/data (286482432 bytes available vs 518351011 bytes reserved) (error 28)
I20260810 22:06:00.858152 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: CompactRowSetsOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.589s user 0.103s sys 0.025s Metrics: {"bytes_written":11550,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5016,"cfile_cache_miss":26,"cfile_cache_miss_bytes":15114,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":32883,"dirs.run_cpu_time_us":2105,"dirs.run_wall_time_us":21345,"drs_written":1,"lbm_read_time_us":3374,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2603,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":3469,"rows_written":263,"thread_start_us":80127,"threads_started":5}
I20260810 22:06:00.869843 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(416ba6bb2af24654a64eb1a23261254b): 2717 bytes on disk
I20260810 22:06:00.874500 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.003s user 0.000s sys 0.001s Metrics: {"cfile_init":1,"lbm_read_time_us":454,"lbm_reads_lt_1ms":4}
W20260810 22:06:00.885435 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.131:44793, user_credentials={real_user=slave}} blocked reactor thread for 56628.1us
I20260810 22:06:00.892582 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(f0037195580e420fad2a77957025fc74): perf score=0.060569
I20260810 22:06:00.967684 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: FlushMRSOp(f0037195580e420fad2a77957025fc74) complete. Timing: real 0.386s user 0.119s sys 0.023s Metrics: {"bytes_written":12135,"cfile_init":1,"dirs.queue_time_us":1532,"dirs.run_cpu_time_us":355,"dirs.run_wall_time_us":971,"drs_written":1,"lbm_read_time_us":269,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2595,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":285}
I20260810 22:06:00.977984 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling FlushMRSOp(416ba6bb2af24654a64eb1a23261254b): perf score=0.030173
W20260810 22:06:01.032449 4596 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-2/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-2/data (252911616 bytes available vs 518351011 bytes reserved) (error 28)
W20260810 22:06:01.264878 5138 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.27.17.130:37933, user_credentials={real_user=slave}} blocked reactor thread for 49233.7us
I20260810 22:06:01.315966 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: CompactRowSetsOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.720s user 0.164s sys 0.111s Metrics: {"bytes_written":11259,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4797,"cfile_cache_miss":26,"cfile_cache_miss_bytes":14517,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":390921,"dirs.run_cpu_time_us":2659,"dirs.run_wall_time_us":35775,"drs_written":1,"lbm_read_time_us":4647,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":2539,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":3310,"rows_written":251,"spinlock_wait_cycles":16256,"thread_start_us":420980,"threads_started":5}
I20260810 22:06:01.337189 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling FlushMRSOp(f0037195580e420fad2a77957025fc74): perf score=0.060569
W20260810 22:06:01.536854 4784 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-1/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-1/data-1/data (185819136 bytes available vs 518351011 bytes reserved) (error 28)
W20260810 22:06:01.745285 4975 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-2/data-0/data (185823232 bytes available vs 518351011 bytes reserved) (error 28)
I20260810 22:06:01.798913 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: FlushMRSOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.820s user 0.075s sys 0.025s Metrics: {"bytes_written":8944,"cfile_init":1,"dirs.queue_time_us":18130,"dirs.run_cpu_time_us":330,"dirs.run_wall_time_us":8230,"drs_written":1,"lbm_read_time_us":4337,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":2970,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":163,"thread_start_us":18330,"threads_started":1}
I20260810 22:06:01.835841 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling CompactRowSetsOp(416ba6bb2af24654a64eb1a23261254b): perf score=1.006900
W20260810 22:06:01.934022 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.129:43093, user_credentials={real_user=slave}} blocked reactor thread for 57699.5us
W20260810 22:06:02.299228 4596 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/data (152170496 bytes available vs 518351011 bytes reserved) (error 28)
I20260810 22:06:02.453586 4784 maintenance_manager.cc:643] P b9235c169d85402faf437d72e9a38bdd: FlushMRSOp(f0037195580e420fad2a77957025fc74) complete. Timing: real 1.098s user 0.125s sys 0.015s Metrics: {"bytes_written":12697,"cfile_init":1,"dirs.queue_time_us":50355,"dirs.run_cpu_time_us":361,"dirs.run_wall_time_us":1208,"drs_written":1,"lbm_read_time_us":280,"lbm_reads_lt_1ms":4,"lbm_write_time_us":26637,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":306,"thread_start_us":19238,"threads_started":1}
I20260810 22:06:02.468039 4940 maintenance_manager.cc:419] P b9235c169d85402faf437d72e9a38bdd: Scheduling FlushMRSOp(df71b8fe94f84062bbf858f1cc4a0920): perf score=0.060569
W20260810 22:06:02.504199 4784 tablet_replica_mm_ops.cc:214] T df71b8fe94f84062bbf858f1cc4a0920 P b9235c169d85402faf437d72e9a38bdd: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet df71b8fe94f84062bbf858f1cc4a0920: unable to open output file for column key INT32 NOT NULL: No directories could be added (3 dirs total, 0 failed, 3 full): IO error: No directories available in df71b8fe94f84062bbf858f1cc4a0920's directory group (3 dirs total, 0 failed, 3 full) (error 28) (error 19)
F20260810 22:06:02.515592 4784 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet df71b8fe94f84062bbf858f1cc4a0920: unable to open output file for column key INT32 NOT NULL: No directories could be added (3 dirs total, 0 failed, 3 full): IO error: No directories available in df71b8fe94f84062bbf858f1cc4a0920's directory group (3 dirs total, 0 failed, 3 full) (error 28) (error 19)
*** Check failure stack trace: ***
I20260810 22:06:02.656333 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(f0037195580e420fad2a77957025fc74) complete. Timing: real 1.763s user 0.118s sys 0.032s Metrics: {"bytes_written":12166,"cfile_init":1,"dirs.queue_time_us":160192,"dirs.run_cpu_time_us":497,"dirs.run_wall_time_us":5017,"drs_written":1,"lbm_read_time_us":380,"lbm_reads_lt_1ms":4,"lbm_write_time_us":20416,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":287,"thread_start_us":146036,"threads_started":1}
I20260810 22:06:02.676877 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(df71b8fe94f84062bbf858f1cc4a0920): perf score=0.060569
W20260810 22:06:02.789559 4596 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskld5GkP/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786399438989427-27718-0/minicluster-data/ts-0/data-0/data (118530048 bytes available vs 518351011 bytes reserved) (error 28)
@ 0x7f7ec8fda9a0 google::LogMessage::Flush() at ??:0
@ 0x7f7ec8fdfadb google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260810 22:06:03.478348 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(df71b8fe94f84062bbf858f1cc4a0920) complete. Timing: real 0.783s user 0.132s sys 0.031s Metrics: {"bytes_written":12486,"cfile_init":1,"dirs.queue_time_us":109510,"dirs.run_cpu_time_us":440,"dirs.run_wall_time_us":45900,"drs_written":1,"lbm_read_time_us":361,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2934,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":297,"thread_start_us":109489,"threads_started":1}
I20260810 22:06:03.487210 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(e33e69aaf4a64216b2f995a9e9e9e1e2): perf score=0.060569
@ 0x7f7ed2517669 kudu::tablet::FlushMRSOp::Perform() at ??:0
I20260810 22:06:03.793776 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: CompactRowSetsOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 1.955s user 0.116s sys 0.052s Metrics: {"bytes_written":11975,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5325,"cfile_cache_miss":26,"cfile_cache_miss_bytes":15993,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":54957,"dirs.run_cpu_time_us":2281,"dirs.run_wall_time_us":24563,"drs_written":1,"lbm_read_time_us":114533,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":3,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2607,"lbm_writes_lt_1ms":23,"mutex_wait_us":23,"num_input_rowsets":2,"peak_mem_usage":3693,"rows_written":280,"thread_start_us":58348,"threads_started":6}
W20260810 22:06:03.796033 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.131:44793, user_credentials={real_user=slave}} blocked reactor thread for 141277us
I20260810 22:06:03.839207 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling UndoDeltaBlockGCOp(f0037195580e420fad2a77957025fc74): 2908 bytes on disk
I20260810 22:06:03.842051 4596 maintenance_manager.cc:643] P 2b27c8ff42ea450bb5114ab97c1ed25d: UndoDeltaBlockGCOp(f0037195580e420fad2a77957025fc74) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":313,"lbm_reads_lt_1ms":4}
I20260810 22:06:03.856515 4752 maintenance_manager.cc:419] P 2b27c8ff42ea450bb5114ab97c1ed25d: Scheduling FlushMRSOp(df71b8fe94f84062bbf858f1cc4a0920): perf score=0.060569
@ 0x7f7ecb0cebd7 kudu::MaintenanceManager::LaunchOp() at ??:0
W20260810 22:06:03.883795 4596 tablet_replica_mm_ops.cc:214] T df71b8fe94f84062bbf858f1cc4a0920 P 2b27c8ff42ea450bb5114ab97c1ed25d: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet df71b8fe94f84062bbf858f1cc4a0920: unable to open output file for column key INT32 NOT NULL: No directories could be added (3 dirs total, 0 failed, 3 full): IO error: No directories available in df71b8fe94f84062bbf858f1cc4a0920's directory group (3 dirs total, 0 failed, 3 full) (error 28) (error 19)
F20260810 22:06:03.884183 4596 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet df71b8fe94f84062bbf858f1cc4a0920: unable to open output file for column key INT32 NOT NULL: No directories could be added (3 dirs total, 0 failed, 3 full): IO error: No directories available in df71b8fe94f84062bbf858f1cc4a0920's directory group (3 dirs total, 0 failed, 3 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7fbf7fb4a9a0 google::LogMessage::Flush() at ??:0
I20260810 22:06:04.133821 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(e33e69aaf4a64216b2f995a9e9e9e1e2) complete. Timing: real 0.646s user 0.117s sys 0.011s Metrics: {"bytes_written":13931,"cfile_init":1,"dirs.queue_time_us":12864,"dirs.run_cpu_time_us":355,"dirs.run_wall_time_us":5172,"drs_written":1,"lbm_read_time_us":271,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2688,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":353,"thread_start_us":19652,"threads_started":1}
@ 0x7fbf7fb4fadb google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260810 22:06:04.165102 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(e33e69aaf4a64216b2f995a9e9e9e1e2): 3550 bytes on disk
I20260810 22:06:04.170859 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(e33e69aaf4a64216b2f995a9e9e9e1e2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":368,"lbm_reads_lt_1ms":4}
@ 0x7fbf89087669 kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7f7ecb0d2b23 kudu::MaintenanceManager::RunSchedulerThread()::$_3::operator()() at ??:0
@ 0x7fbf81c3ebd7 kudu::MaintenanceManager::LaunchOp() at ??:0
I20260810 22:06:04.177230 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(df71b8fe94f84062bbf858f1cc4a0920): 3043 bytes on disk
I20260810 22:06:04.230691 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(df71b8fe94f84062bbf858f1cc4a0920) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":270,"lbm_reads_lt_1ms":4}
@ 0x7fbf81c42b23 kudu::MaintenanceManager::RunSchedulerThread()::$_3::operator()() at ??:0
@ 0x7f7ecb0d2aca _ZNSt3__18__invokeIRZN4kudu18MaintenanceManager18RunSchedulerThreadEvE3$_3JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
@ 0x7fbf81c42aca _ZNSt3__18__invokeIRZN4kudu18MaintenanceManager18RunSchedulerThreadEvE3$_3JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
I20260810 22:06:04.296371 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling CompactRowSetsOp(e33e69aaf4a64216b2f995a9e9e9e1e2): perf score=0.846717
@ 0x7fbf81c42a5a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
W20260810 22:06:01.283404 5300 meta_cache.cc:1261] Time spent looking up entry by key: real 0.062s user 0.003s sys 0.000s
@ 0x7f7ecb0d2a5a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7f7ecb0d2a22 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7f7ecb0d1d1e std::__1::__function::__func<>::operator()() at ??:0
@ 0x580785 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x7fbf81c42a22 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x5805ba std::__1::function<>::operator()() at ??:?
@ 0x7f7ecb1a96b7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7ecb1acb12 kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
W20260810 22:06:04.683772 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.129:43093, user_credentials={real_user=slave}} blocked reactor thread for 49868.4us
@ 0x7f7ecb1acaca _ZNSt3__18__invokeIRZN4kudu10ThreadPool12CreateThreadEvE3$_2JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
@ 0x7f7ecb1aca5a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
W20260810 22:06:04.936522 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.129:43093, user_credentials={real_user=slave}} blocked reactor thread for 67735.8us
@ 0x7fbf81c41d1e std::__1::__function::__func<>::operator()() at ??:0
@ 0x580785 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5805ba std::__1::function<>::operator()() at ??:?
@ 0x7fbf81d196b7 kudu::ThreadPool::DispatchThread() at ??:0
W20260810 22:06:05.285528 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.129:43093, user_credentials={real_user=slave}} blocked reactor thread for 93739.8us
@ 0x7fbf81d1cb12 kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7ecb1aca22 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7fbf81d1caca _ZNSt3__18__invokeIRZN4kudu10ThreadPool12CreateThreadEvE3$_2JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
@ 0x7f7ecb1abd1e std::__1::__function::__func<>::operator()() at ??:0
@ 0x580785 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5805ba std::__1::function<>::operator()() at ??:?
@ 0x7fbf81d1ca5a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
W20260810 22:06:06.081844 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.129:43093, user_credentials={real_user=slave}} blocked reactor thread for 81624.1us
@ 0x7fbf81d1ca22 std::__1::__function::__alloc_func<>::operator()() at ??:0
W20260810 22:06:06.184409 4591 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.131:44793, user_credentials={real_user=slave}} blocked reactor thread for 46151.3us
@ 0x7f7ecb18bdc7 kudu::Thread::SuperviseThread() at ??:0
@ 0x4230f0 __tsan_thread_start_func at ??:?
I20260810 22:06:06.315409 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: CompactRowSetsOp(e33e69aaf4a64216b2f995a9e9e9e1e2) complete. Timing: real 1.965s user 0.105s sys 0.035s Metrics: {"bytes_written":14256,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6931,"cfile_cache_miss":26,"cfile_cache_miss_bytes":20557,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":416160,"dirs.run_cpu_time_us":2895,"dirs.run_wall_time_us":118870,"drs_written":1,"lbm_read_time_us":163969,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":43,"lbm_write_time_us":2548,"lbm_writes_lt_1ms":23,"mutex_wait_us":25,"num_input_rowsets":2,"peak_mem_usage":4979,"rows_written":368,"spinlock_wait_cycles":113920,"thread_start_us":495085,"threads_started":6}
I20260810 22:06:06.325966 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(628379a2077c4ea7896784b089a5ea0c): perf score=0.060569
@ 0x7fbf81d1bd1e std::__1::__function::__func<>::operator()() at ??:0
@ 0x580785 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5805ba std::__1::function<>::operator()() at ??:?
@ 0x7fbf81cfbdc7 kudu::Thread::SuperviseThread() at ??:0
@ 0x4230f0 __tsan_thread_start_func at ??:?
@ 0x7fbf833696db start_thread at ??:0
@ 0x7fbf7e7ef61f clone at ??:0
@ 0x7f7ecc7f96db start_thread at ??:0
W20260810 22:06:06.788899 4781 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.17.131:44793, user_credentials={real_user=slave}} blocked reactor thread for 63595.8us
@ 0x7f7ec7c7f61f clone at ??:0
I20260810 22:06:06.946328 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(628379a2077c4ea7896784b089a5ea0c) complete. Timing: real 0.620s user 0.121s sys 0.017s Metrics: {"bytes_written":13352,"cfile_init":1,"dirs.queue_time_us":975,"dirs.run_cpu_time_us":339,"dirs.run_wall_time_us":1441,"drs_written":1,"lbm_read_time_us":270,"lbm_reads_lt_1ms":4,"lbm_write_time_us":53349,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":332,"thread_start_us":9943,"threads_started":1}
I20260810 22:06:06.953270 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling CompactRowSetsOp(628379a2077c4ea7896784b089a5ea0c): perf score=1.010531
I20260810 22:06:07.229436 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: CompactRowSetsOp(628379a2077c4ea7896784b089a5ea0c) complete. Timing: real 0.275s user 0.146s sys 0.032s Metrics: {"bytes_written":15419,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7713,"cfile_cache_miss":26,"cfile_cache_miss_bytes":22839,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":7133,"dirs.run_cpu_time_us":2317,"dirs.run_wall_time_us":9126,"drs_written":1,"lbm_read_time_us":4111,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2992,"lbm_writes_lt_1ms":23,"mutex_wait_us":37,"num_input_rowsets":2,"peak_mem_usage":5516,"rows_written":411,"spinlock_wait_cycles":42752,"thread_start_us":52753,"threads_started":3}
I20260810 22:06:07.237586 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(778d8c794af64277b2f497708012e16c): perf score=0.060569
W20260810 22:06:07.330899 4971 connection.cc:535] server connection from 127.27.17.130:54991 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 22:06:07.331733 5138 connection.cc:535] client connection to 127.27.17.130:37933 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 22:06:07.332335 5138 meta_cache.cc:302] tablet 628379a2077c4ea7896784b089a5ea0c: replica b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933) has failed: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 22:06:07.394208 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(778d8c794af64277b2f497708012e16c) complete. Timing: real 0.153s user 0.105s sys 0.009s Metrics: {"bytes_written":12314,"cfile_init":1,"dirs.queue_time_us":106,"dirs.run_cpu_time_us":291,"dirs.run_wall_time_us":1413,"drs_written":1,"lbm_read_time_us":270,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2540,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":291}
I20260810 22:06:07.400861 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(628379a2077c4ea7896784b089a5ea0c): 4064 bytes on disk
W20260810 22:06:07.407063 5001 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.408974 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.411628 5137 connection.cc:535] client connection to 127.27.17.129:43093 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 22:06:07.405217 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:07.414427 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(628379a2077c4ea7896784b089a5ea0c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":288,"lbm_reads_lt_1ms":4}
I20260810 22:06:07.420596 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling CompactRowSetsOp(778d8c794af64277b2f497708012e16c): perf score=1.009346
W20260810 22:06:07.429037 5922 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:06:07.414458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.17.129:43093 (local address 127.0.0.1:40616)
0810 22:06:07.415001 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0810 22:06:07.415043 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 22:06:07.415294 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:06:07.427234 (+ 11940us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":87}
W20260810 22:06:07.426620 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.427935 5001 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.435561 4996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.429659 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.439980 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.431155 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.443038 4993 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.457064 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.465585 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.471639 4992 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.479633 4989 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.473556 4988 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.482753 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.486939 4994 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.486645 4996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.487510 4993 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.542438 5001 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.550741 5001 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.556200 5001 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.559324 4993 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.562251 4993 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.567785 5299 meta_cache.cc:1261] Time spent looking up entry by key: real 0.070s user 0.000s sys 0.000s
W20260810 22:06:07.570850 4993 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.581060 4993 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.584538 4993 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.590744 4993 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.634982 4993 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.641441 4993 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.644548 5001 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.650184 4993 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.655735 5001 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:07.710461 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: CompactRowSetsOp(778d8c794af64277b2f497708012e16c) complete. Timing: real 0.289s user 0.155s sys 0.058s Metrics: {"bytes_written":15453,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7752,"cfile_cache_miss":26,"cfile_cache_miss_bytes":23182,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":19828,"dirs.run_cpu_time_us":2235,"dirs.run_wall_time_us":26168,"drs_written":1,"lbm_read_time_us":3504,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2597,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":5504,"rows_written":413,"thread_start_us":103305,"threads_started":4}
I20260810 22:06:07.717165 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(416ba6bb2af24654a64eb1a23261254b): perf score=0.030173
W20260810 22:06:07.795966 4993 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:07.801240 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.083s user 0.070s sys 0.008s Metrics: {"bytes_written":8776,"cfile_init":1,"dirs.queue_time_us":87,"dirs.run_cpu_time_us":278,"dirs.run_wall_time_us":1429,"drs_written":1,"lbm_read_time_us":269,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2405,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":157,"thread_start_us":10623,"threads_started":1,"wal-append.queue_time_us":10985}
W20260810 22:06:07.805117 5001 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:07.807550 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(778d8c794af64277b2f497708012e16c): 4101 bytes on disk
W20260810 22:06:07.808849 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:07.809304 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(778d8c794af64277b2f497708012e16c) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":293,"lbm_reads_lt_1ms":4,"spinlock_wait_cycles":66816}
W20260810 22:06:07.807572 4996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:07.814411 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling CompactRowSetsOp(416ba6bb2af24654a64eb1a23261254b): perf score=1.008398
W20260810 22:06:07.815647 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.818060 4996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.819727 5001 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.821400 4993 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.826663 4992 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.823050 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.828739 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.829631 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.830840 4988 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.825330 4994 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.831995 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.836696 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.833657 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.834311 4996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.840595 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.841297 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.843464 4996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.845760 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.848655 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.852272 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.853643 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.854259 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.858209 4996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.864359 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:07.871232 5933 raft_consensus.cc:482] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:07.871933 5933 raft_consensus.cc:504] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
W20260810 22:06:07.879848 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:07.883680 5933 leader_election.cc:288] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:07.888571 4974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:07.902082 5933 raft_consensus.cc:482] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:07.902631 5933 raft_consensus.cc:504] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:07.904557 5933 leader_election.cc:288] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:07.907261 4971 leader_election.cc:334] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:07.909708 4971 leader_election.cc:334] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:07.910550 4974 leader_election.cc:334] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:07.911080 4974 leader_election.cc:302] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:07.911952 5933 raft_consensus.cc:2726] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:07.932986 4974 leader_election.cc:334] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:07.934026 4974 leader_election.cc:302] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:07.935014 5933 raft_consensus.cc:2726] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:06:07.946846 5933 raft_consensus.cc:482] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:07.947345 5933 raft_consensus.cc:504] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } } peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } }
I20260810 22:06:07.950018 5933 leader_election.cc:288] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933), 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093)
W20260810 22:06:07.961525 4971 leader_election.cc:334] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:07.962518 4974 leader_election.cc:334] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:07.963016 4974 leader_election.cc:302] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:07.966543 5933 raft_consensus.cc:2726] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:06:07.967139 5933 raft_consensus.cc:482] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:07.968750 5933 raft_consensus.cc:504] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
W20260810 22:06:07.977097 4971 leader_election.cc:334] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
I20260810 22:06:07.978077 5933 leader_election.cc:288] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:07.985152 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.986706 4974 leader_election.cc:334] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:07.987195 4974 leader_election.cc:302] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:07.988284 5933 raft_consensus.cc:2726] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:06:07.995713 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: CompactRowSetsOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.181s user 0.132s sys 0.008s Metrics: {"bytes_written":15676,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7881,"cfile_cache_miss":26,"cfile_cache_miss_bytes":23485,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":1705,"dirs.run_cpu_time_us":2022,"dirs.run_wall_time_us":8759,"drs_written":1,"lbm_read_time_us":3601,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2746,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":5570,"rows_written":420,"spinlock_wait_cycles":176512,"thread_start_us":6358,"threads_started":1}
W20260810 22:06:07.998680 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:07.999470 4996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:08.002812 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(416ba6bb2af24654a64eb1a23261254b): 4174 bytes on disk
W20260810 22:06:08.004021 4996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:08.006153 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(416ba6bb2af24654a64eb1a23261254b) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":292,"lbm_reads_lt_1ms":4}
W20260810 22:06:08.012434 4996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:08.015008 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(e33e69aaf4a64216b2f995a9e9e9e1e2): 3685 bytes on disk
W20260810 22:06:08.015110 4996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.015636 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.015959 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:08.016650 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(e33e69aaf4a64216b2f995a9e9e9e1e2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":238,"lbm_reads_lt_1ms":4}
W20260810 22:06:08.018345 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.018707 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:08.021484 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(518b19decff74b1baf253900da181d46): perf score=0.030173
W20260810 22:06:08.023501 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.025950 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.028306 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:08.034708 5933 raft_consensus.cc:482] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:08.035171 5933 raft_consensus.cc:504] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:08.037040 5933 leader_election.cc:288] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
I20260810 22:06:08.038199 5933 raft_consensus.cc:482] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:08.038722 5933 raft_consensus.cc:504] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:08.040670 5933 leader_election.cc:288] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:08.042490 4971 leader_election.cc:334] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:08.042915 4974 leader_election.cc:334] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:08.043409 4974 leader_election.cc:302] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
W20260810 22:06:08.045609 4971 leader_election.cc:334] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
I20260810 22:06:08.045755 5933 raft_consensus.cc:2726] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:08.046825 4974 leader_election.cc:334] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:08.047214 4974 leader_election.cc:302] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:08.047762 5933 raft_consensus.cc:482] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:08.048400 5937 raft_consensus.cc:2726] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:06:08.048668 5933 raft_consensus.cc:504] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:08.050653 5933 leader_election.cc:288] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
I20260810 22:06:08.054790 5933 raft_consensus.cc:482] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:08.056104 5933 raft_consensus.cc:504] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:08.058802 5933 leader_election.cc:288] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:08.055682 4971 leader_election.cc:334] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:08.059629 4974 leader_election.cc:334] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:08.060951 4974 leader_election.cc:302] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
W20260810 22:06:08.062165 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.064692 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.066982 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.069694 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.076151 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.076663 4996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.077865 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.078636 4974 leader_election.cc:334] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:08.079075 5933 raft_consensus.cc:2726] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:08.080091 4971 leader_election.cc:334] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
I20260810 22:06:08.080513 4971 leader_election.cc:302] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:08.081266 5933 raft_consensus.cc:2726] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:06:08.090972 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(518b19decff74b1baf253900da181d46) complete. Timing: real 0.069s user 0.054s sys 0.012s Metrics: {"bytes_written":9584,"cfile_init":1,"dirs.queue_time_us":94,"dirs.run_cpu_time_us":263,"dirs.run_wall_time_us":1427,"drs_written":1,"lbm_read_time_us":261,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2327,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":187,"thread_start_us":1018,"threads_started":1,"wal-append.queue_time_us":1371}
I20260810 22:06:08.098623 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(f0037195580e420fad2a77957025fc74): 2927 bytes on disk
I20260810 22:06:08.100158 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(f0037195580e420fad2a77957025fc74) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":203,"lbm_reads_lt_1ms":4}
I20260810 22:06:08.105007 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling CompactRowSetsOp(518b19decff74b1baf253900da181d46): perf score=1.012997
W20260810 22:06:08.141245 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.144282 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.148658 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.151191 4996 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.152105 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.152614 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.154563 4994 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.156739 4988 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.158767 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.162788 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.164659 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.169446 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.171793 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.217300 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:08.224313 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: CompactRowSetsOp(518b19decff74b1baf253900da181d46) complete. Timing: real 0.119s user 0.105s sys 0.004s Metrics: {"bytes_written":16240,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8279,"cfile_cache_miss":26,"cfile_cache_miss_bytes":24651,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":2706,"dirs.run_cpu_time_us":2236,"dirs.run_wall_time_us":11820,"drs_written":1,"lbm_read_time_us":2759,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2435,"lbm_writes_lt_1ms":23,"mutex_wait_us":605,"num_input_rowsets":2,"peak_mem_usage":5851,"rows_written":442,"spinlock_wait_cycles":26496,"thread_start_us":2898,"threads_started":3}
I20260810 22:06:08.230542 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(8e619eaa340248da8a54a913c1d16b9f): perf score=0.030173
W20260810 22:06:08.256407 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.257704 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.259920 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.261543 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.266559 5002 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.267483 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.268620 4988 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.269384 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.270713 4994 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.272954 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.275803 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.279134 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.283397 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:08.300012 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(8e619eaa340248da8a54a913c1d16b9f) complete. Timing: real 0.068s user 0.063s sys 0.001s Metrics: {"bytes_written":9433,"cfile_init":1,"dirs.queue_time_us":90,"dirs.run_cpu_time_us":280,"dirs.run_wall_time_us":1423,"drs_written":1,"lbm_read_time_us":250,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2322,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":182,"thread_start_us":574,"threads_started":1,"wal-append.queue_time_us":899}
I20260810 22:06:08.306483 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling CompactRowSetsOp(8e619eaa340248da8a54a913c1d16b9f): perf score=1.009912
I20260810 22:06:08.310966 5933 raft_consensus.cc:482] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:08.311482 5933 raft_consensus.cc:504] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:08.313446 5933 leader_election.cc:288] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:08.320089 4971 leader_election.cc:334] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:08.320771 4974 leader_election.cc:334] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:08.321192 4974 leader_election.cc:302] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:08.321501 5933 raft_consensus.cc:482] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:08.321921 5937 raft_consensus.cc:2726] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:06:08.322006 5933 raft_consensus.cc:504] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:08.323838 5933 leader_election.cc:288] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:08.328085 4971 leader_election.cc:334] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:08.329380 4974 leader_election.cc:334] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:08.329757 4974 leader_election.cc:302] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:08.330483 5933 raft_consensus.cc:2726] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:08.332320 5137 meta_cache.cc:302] tablet 416ba6bb2af24654a64eb1a23261254b: replica 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093) has failed: Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:08.349037 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.349941 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.352550 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.355257 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.355304 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.369939 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.374320 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.407900 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:08.410075 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: CompactRowSetsOp(8e619eaa340248da8a54a913c1d16b9f) complete. Timing: real 0.103s user 0.098s sys 0.002s Metrics: {"bytes_written":15002,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7425,"cfile_cache_miss":26,"cfile_cache_miss_bytes":22181,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":2524,"dirs.run_cpu_time_us":2295,"dirs.run_wall_time_us":10277,"drs_written":1,"lbm_read_time_us":3422,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2306,"lbm_writes_lt_1ms":23,"mutex_wait_us":415,"num_input_rowsets":2,"peak_mem_usage":5218,"rows_written":395,"spinlock_wait_cycles":74496,"thread_start_us":1899,"threads_started":2}
W20260810 22:06:08.412191 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:08.415387 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(af1fb81326b54dbbb23c7b577bdaf751): perf score=0.030173
W20260810 22:06:08.415944 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.422132 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.428387 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.430389 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.431473 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.472515 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.476739 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.477208 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.478753 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:08.478963 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(af1fb81326b54dbbb23c7b577bdaf751) complete. Timing: real 0.063s user 0.060s sys 0.001s Metrics: {"bytes_written":9425,"cfile_init":1,"dirs.queue_time_us":97,"dirs.run_cpu_time_us":347,"dirs.run_wall_time_us":1501,"drs_written":1,"lbm_read_time_us":210,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2185,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":182,"thread_start_us":763,"threads_started":1,"wal-append.queue_time_us":1200}
W20260810 22:06:08.482496 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.482496 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.483139 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:08.485270 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling CompactRowSetsOp(af1fb81326b54dbbb23c7b577bdaf751): perf score=0.994555
W20260810 22:06:08.509878 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.556222 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.556695 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.559155 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.562605 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:08.586400 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: CompactRowSetsOp(af1fb81326b54dbbb23c7b577bdaf751) complete. Timing: real 0.100s user 0.096s sys 0.001s Metrics: {"bytes_written":16023,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8133,"cfile_cache_miss":26,"cfile_cache_miss_bytes":24213,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":2561,"dirs.run_cpu_time_us":2191,"dirs.run_wall_time_us":8824,"drs_written":1,"lbm_read_time_us":2663,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2216,"lbm_writes_lt_1ms":23,"mutex_wait_us":586,"num_input_rowsets":2,"peak_mem_usage":5742,"rows_written":434,"spinlock_wait_cycles":87168,"thread_start_us":747,"threads_started":1}
I20260810 22:06:08.591838 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(dc4801ac5aeb4b4d8340ee2f5c0d73f6): perf score=0.030173
W20260810 22:06:08.608151 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.609645 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.612272 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.628886 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.633142 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.634332 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.636484 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.638063 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.639413 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.642464 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.644381 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:08.664713 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(dc4801ac5aeb4b4d8340ee2f5c0d73f6) complete. Timing: real 0.072s user 0.067s sys 0.000s Metrics: {"bytes_written":11033,"cfile_init":1,"dirs.queue_time_us":94,"dirs.run_cpu_time_us":293,"dirs.run_wall_time_us":1375,"drs_written":1,"lbm_read_time_us":280,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2437,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":242,"thread_start_us":608,"threads_started":1,"wal-append.queue_time_us":897}
I20260810 22:06:08.670346 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(518b19decff74b1baf253900da181d46): 4345 bytes on disk
I20260810 22:06:08.671911 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(518b19decff74b1baf253900da181d46) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":213,"lbm_reads_lt_1ms":4}
I20260810 22:06:08.676422 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(af1fb81326b54dbbb23c7b577bdaf751): 4292 bytes on disk
I20260810 22:06:08.684178 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(af1fb81326b54dbbb23c7b577bdaf751) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":291,"lbm_reads_lt_1ms":4}
I20260810 22:06:08.688526 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling CompactRowSetsOp(dc4801ac5aeb4b4d8340ee2f5c0d73f6): perf score=1.008346
W20260810 22:06:08.694012 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.694965 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.717635 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.719161 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.767741 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.773838 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.778440 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.784502 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:08.792271 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: CompactRowSetsOp(dc4801ac5aeb4b4d8340ee2f5c0d73f6) complete. Timing: real 0.103s user 0.096s sys 0.003s Metrics: {"bytes_written":16012,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8116,"cfile_cache_miss":26,"cfile_cache_miss_bytes":24178,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":1681,"dirs.run_cpu_time_us":1980,"dirs.run_wall_time_us":11754,"drs_written":1,"lbm_read_time_us":2729,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2385,"lbm_writes_lt_1ms":23,"mutex_wait_us":453,"num_input_rowsets":2,"peak_mem_usage":5720,"rows_written":433,"spinlock_wait_cycles":269568,"thread_start_us":548,"threads_started":1}
I20260810 22:06:08.796640 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(dc4801ac5aeb4b4d8340ee2f5c0d73f6): 4264 bytes on disk
I20260810 22:06:08.798144 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(dc4801ac5aeb4b4d8340ee2f5c0d73f6) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":268,"lbm_reads_lt_1ms":4}
I20260810 22:06:08.802299 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(df71b8fe94f84062bbf858f1cc4a0920): perf score=0.014976
W20260810 22:06:08.805087 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.819628 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.843744 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:08.845923 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(df71b8fe94f84062bbf858f1cc4a0920) complete. Timing: real 0.043s user 0.042s sys 0.000s Metrics: {"bytes_written":6925,"cfile_init":1,"dirs.queue_time_us":96,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":1368,"drs_written":1,"lbm_read_time_us":162,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2088,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":87,"thread_start_us":519,"threads_started":1,"wal-append.queue_time_us":843}
I20260810 22:06:08.850920 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling CompactRowSetsOp(df71b8fe94f84062bbf858f1cc4a0920): perf score=1.011575
W20260810 22:06:08.851032 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.853308 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.860231 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.860498 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.864486 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.867354 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.871094 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.897888 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.922832 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.926501 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:08.934922 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: CompactRowSetsOp(df71b8fe94f84062bbf858f1cc4a0920) complete. Timing: real 0.083s user 0.071s sys 0.012s Metrics: {"bytes_written":14666,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7224,"cfile_cache_miss":26,"cfile_cache_miss_bytes":21608,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":2471,"dirs.run_cpu_time_us":2118,"dirs.run_wall_time_us":8058,"drs_written":1,"lbm_read_time_us":3205,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2022,"lbm_writes_lt_1ms":23,"mutex_wait_us":591,"num_input_rowsets":2,"peak_mem_usage":5127,"rows_written":384,"spinlock_wait_cycles":577408,"thread_start_us":560,"threads_started":1}
I20260810 22:06:08.939061 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(8e619eaa340248da8a54a913c1d16b9f): 3967 bytes on disk
I20260810 22:06:08.940292 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(8e619eaa340248da8a54a913c1d16b9f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":175,"lbm_reads_lt_1ms":4}
I20260810 22:06:08.944684 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(f0037195580e420fad2a77957025fc74): perf score=0.014976
W20260810 22:06:08.949770 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.964304 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.968813 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.975459 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:08.990870 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:08.991750 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(f0037195580e420fad2a77957025fc74) complete. Timing: real 0.046s user 0.044s sys 0.000s Metrics: {"bytes_written":8287,"cfile_init":1,"dirs.queue_time_us":148,"dirs.run_cpu_time_us":280,"dirs.run_wall_time_us":1775,"drs_written":1,"lbm_read_time_us":260,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2363,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":137,"thread_start_us":407,"threads_started":1,"wal-append.queue_time_us":699}
I20260810 22:06:08.996055 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(df71b8fe94f84062bbf858f1cc4a0920): 3865 bytes on disk
I20260810 22:06:08.997231 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(df71b8fe94f84062bbf858f1cc4a0920) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":192,"lbm_reads_lt_1ms":4}
I20260810 22:06:09.000327 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling CompactRowSetsOp(f0037195580e420fad2a77957025fc74): perf score=1.009193
W20260810 22:06:09.022089 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.065043 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.087456 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.090371 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:09.105402 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: CompactRowSetsOp(f0037195580e420fad2a77957025fc74) complete. Timing: real 0.104s user 0.098s sys 0.004s Metrics: {"bytes_written":15735,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7954,"cfile_cache_miss":26,"cfile_cache_miss_bytes":23710,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":2588,"dirs.run_cpu_time_us":1851,"dirs.run_wall_time_us":6911,"drs_written":1,"lbm_read_time_us":3379,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2122,"lbm_writes_lt_1ms":23,"mutex_wait_us":307,"num_input_rowsets":2,"peak_mem_usage":5647,"rows_written":424,"spinlock_wait_cycles":163072,"thread_start_us":985,"threads_started":1}
I20260810 22:06:09.109688 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(f0037195580e420fad2a77957025fc74): 4195 bytes on disk
I20260810 22:06:09.111152 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(f0037195580e420fad2a77957025fc74) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":278,"lbm_reads_lt_1ms":4}
W20260810 22:06:09.114691 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:09.115626 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(e33e69aaf4a64216b2f995a9e9e9e1e2): perf score=0.007377
W20260810 22:06:09.115581 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.132377 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.135841 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.138965 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.149771 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:09.163089 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(e33e69aaf4a64216b2f995a9e9e9e1e2) complete. Timing: real 0.046s user 0.035s sys 0.007s Metrics: {"bytes_written":6466,"cfile_init":1,"dirs.queue_time_us":1024,"dirs.run_cpu_time_us":333,"dirs.run_wall_time_us":1786,"drs_written":1,"lbm_read_time_us":293,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2493,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":69,"thread_start_us":543,"threads_started":1,"wal-append.queue_time_us":927}
W20260810 22:06:09.167438 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:09.168607 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(e33e69aaf4a64216b2f995a9e9e9e1e2): 805 bytes on disk
I20260810 22:06:09.170375 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(e33e69aaf4a64216b2f995a9e9e9e1e2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":329,"lbm_reads_lt_1ms":4}
I20260810 22:06:09.175172 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling CompactRowSetsOp(e33e69aaf4a64216b2f995a9e9e9e1e2): perf score=0.975989
W20260810 22:06:09.191702 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.203418 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.203982 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.251168 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.271344 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.273839 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:09.274731 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: CompactRowSetsOp(e33e69aaf4a64216b2f995a9e9e9e1e2) complete. Timing: real 0.099s user 0.095s sys 0.001s Metrics: {"bytes_written":16072,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8190,"cfile_cache_miss":26,"cfile_cache_miss_bytes":24204,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":3189,"dirs.run_cpu_time_us":1402,"dirs.run_wall_time_us":8407,"drs_written":1,"lbm_read_time_us":2513,"lbm_reads_lt_1ms":46,"lbm_write_time_us":1950,"lbm_writes_lt_1ms":23,"mutex_wait_us":247,"num_input_rowsets":2,"peak_mem_usage":5899,"rows_written":437,"spinlock_wait_cycles":37888}
I20260810 22:06:09.280138 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(e33e69aaf4a64216b2f995a9e9e9e1e2): 4329 bytes on disk
I20260810 22:06:09.281461 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(e33e69aaf4a64216b2f995a9e9e9e1e2) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":186,"lbm_reads_lt_1ms":4}
I20260810 22:06:09.284479 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling FlushMRSOp(628379a2077c4ea7896784b089a5ea0c): perf score=0.003577
W20260810 22:06:09.303071 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.304761 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.307335 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.314631 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.315954 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:09.315541 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: FlushMRSOp(628379a2077c4ea7896784b089a5ea0c) complete. Timing: real 0.030s user 0.026s sys 0.003s Metrics: {"bytes_written":5178,"cfile_init":1,"dirs.queue_time_us":93,"dirs.run_cpu_time_us":351,"dirs.run_wall_time_us":1407,"drs_written":1,"lbm_read_time_us":159,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1749,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":21,"thread_start_us":844,"threads_started":1,"wal-append.queue_time_us":1230}
I20260810 22:06:09.321271 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(628379a2077c4ea7896784b089a5ea0c): 315 bytes on disk
I20260810 22:06:09.322768 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(628379a2077c4ea7896784b089a5ea0c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":206,"lbm_reads_lt_1ms":4}
W20260810 22:06:09.326807 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:09.327221 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling CompactRowSetsOp(628379a2077c4ea7896784b089a5ea0c): perf score=0.884141
W20260810 22:06:09.326754 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.333694 5138 meta_cache.cc:302] tablet f0037195580e420fad2a77957025fc74: replica b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933) has failed: Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
W20260810 22:06:09.360013 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.380579 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.387356 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.391204 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.410600 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.415874 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:09.419566 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: CompactRowSetsOp(628379a2077c4ea7896784b089a5ea0c) complete. Timing: real 0.092s user 0.086s sys 0.004s Metrics: {"bytes_written":15929,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8098,"cfile_cache_miss":26,"cfile_cache_miss_bytes":23826,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":1670,"dirs.run_cpu_time_us":1506,"dirs.run_wall_time_us":7796,"drs_written":1,"lbm_read_time_us":2382,"lbm_reads_lt_1ms":46,"lbm_write_time_us":2360,"lbm_writes_lt_1ms":23,"mutex_wait_us":369,"num_input_rowsets":2,"peak_mem_usage":5879,"rows_written":432}
I20260810 22:06:09.426015 5125 maintenance_manager.cc:419] P a552c4b7a11b4cb7a2fdaacbcdf219dc: Scheduling UndoDeltaBlockGCOp(628379a2077c4ea7896784b089a5ea0c): 4252 bytes on disk
I20260810 22:06:09.427258 4975 maintenance_manager.cc:643] P a552c4b7a11b4cb7a2fdaacbcdf219dc: UndoDeltaBlockGCOp(628379a2077c4ea7896784b089a5ea0c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":189,"lbm_reads_lt_1ms":4}
W20260810 22:06:09.451076 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.453034 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.485612 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:09.493645 5966 raft_consensus.cc:482] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:09.494112 5966 raft_consensus.cc:504] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:09.495795 5966 leader_election.cc:288] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:09.503408 4971 leader_election.cc:334] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:09.503928 4974 leader_election.cc:334] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:09.504328 4974 leader_election.cc:302] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:09.504963 5966 raft_consensus.cc:2726] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:09.516971 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.562120 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.563117 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.588013 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.620383 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.645390 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.646591 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:09.652930 5966 raft_consensus.cc:482] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:09.653434 5966 raft_consensus.cc:504] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } } peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } }
I20260810 22:06:09.655110 5966 leader_election.cc:288] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933), 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093)
W20260810 22:06:09.658679 4974 leader_election.cc:334] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
W20260810 22:06:09.659184 4971 leader_election.cc:334] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
I20260810 22:06:09.659543 4971 leader_election.cc:302] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:09.660225 5966 raft_consensus.cc:2726] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:09.668409 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.670125 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:09.671552 5966 raft_consensus.cc:482] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:09.671972 5966 raft_consensus.cc:504] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:09.673620 5966 leader_election.cc:288] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:09.677621 4974 leader_election.cc:334] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
W20260810 22:06:09.677963 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.679064 4971 leader_election.cc:334] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
I20260810 22:06:09.679459 4971 leader_election.cc:302] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:09.680124 5966 raft_consensus.cc:2726] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:09.682888 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.682888 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.686259 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.687597 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.689708 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:09.691692 5966 raft_consensus.cc:482] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:09.692147 5966 raft_consensus.cc:504] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:09.694226 5966 leader_election.cc:288] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:09.699921 4971 leader_election.cc:334] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:09.700470 4974 leader_election.cc:334] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:09.700887 4974 leader_election.cc:302] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:09.701632 5966 raft_consensus.cc:2726] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:09.702257 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.702776 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.762753 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.765394 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.767182 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.773607 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.774098 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.776733 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.777952 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.835948 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:09.853577 5966 raft_consensus.cc:482] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:09.854043 5966 raft_consensus.cc:504] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:09.855624 5966 leader_election.cc:288] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:09.858954 4971 leader_election.cc:334] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:09.859393 4974 leader_election.cc:334] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:09.859850 4974 leader_election.cc:302] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:09.860687 5966 raft_consensus.cc:2726] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:09.877955 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.899135 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.902292 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.902834 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.903337 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.949329 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:09.968322 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:10.005385 5966 raft_consensus.cc:482] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:10.005998 5966 raft_consensus.cc:504] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:10.007746 5966 leader_election.cc:288] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:10.011246 4971 leader_election.cc:334] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:10.011657 4974 leader_election.cc:334] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:10.012053 4974 leader_election.cc:302] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:10.012676 5966 raft_consensus.cc:2726] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:10.035171 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.037236 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.089757 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:10.094688 5966 raft_consensus.cc:482] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:10.095247 5966 raft_consensus.cc:504] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
W20260810 22:06:10.095662 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:10.097033 5966 leader_election.cc:288] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:10.100958 4971 leader_election.cc:334] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:10.101464 4974 leader_election.cc:334] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:10.101863 4974 leader_election.cc:302] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:10.105535 5966 raft_consensus.cc:2726] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:06:10.128057 5966 raft_consensus.cc:482] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:10.128568 5966 raft_consensus.cc:504] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:10.130515 5966 leader_election.cc:288] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:10.135182 4971 leader_election.cc:334] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:10.135680 4974 leader_election.cc:334] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:10.136093 4974 leader_election.cc:302] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:10.136794 5966 raft_consensus.cc:2726] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:06:10.140745 5966 raft_consensus.cc:482] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:10.141146 5966 raft_consensus.cc:504] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:10.142740 5966 leader_election.cc:288] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:10.146098 4971 leader_election.cc:334] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:10.147521 4974 leader_election.cc:334] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:10.147943 4974 leader_election.cc:302] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:10.148576 5966 raft_consensus.cc:2726] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:06:10.167521 5966 raft_consensus.cc:482] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:10.168084 5966 raft_consensus.cc:504] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:10.170037 5966 leader_election.cc:288] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:10.173471 4971 leader_election.cc:334] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:10.173928 4974 leader_election.cc:334] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:10.174327 4974 leader_election.cc:302] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:10.175343 5966 raft_consensus.cc:2726] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:10.178227 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.178229 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.184015 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.191407 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.194226 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.194906 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.206929 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.214846 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.215636 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.225601 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.245209 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.298756 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.312485 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.315647 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.334964 5138 meta_cache.cc:302] tablet e33e69aaf4a64216b2f995a9e9e9e1e2: replica b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933) has failed: Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
W20260810 22:06:10.346342 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.438462 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.448729 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.449968 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.457734 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.458163 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.477509 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.480434 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.491358 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.593618 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.596565 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.619112 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.663782 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.669929 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.696591 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.749630 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.750700 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.752177 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.757602 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.762025 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.827286 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.863576 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.867053 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.890128 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.897842 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.902259 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.903249 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.905277 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.905925 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:10.908794 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.015722 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.045260 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.064289 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.065305 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.071986 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.096634 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.104357 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.203387 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.205750 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.216275 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.216341 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.225564 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.306197 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.335577 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.338008 5137 meta_cache.cc:302] tablet e33e69aaf4a64216b2f995a9e9e9e1e2: replica 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093) has failed: Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:11.349673 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.361373 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.362999 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.385114 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.386646 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.417173 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.441784 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:11.449772 5974 raft_consensus.cc:482] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:11.450275 5974 raft_consensus.cc:504] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:11.452221 5974 leader_election.cc:288] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:11.458878 4971 leader_election.cc:334] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:11.459779 4974 leader_election.cc:334] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:11.460176 4974 leader_election.cc:302] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:11.460812 5974 raft_consensus.cc:2726] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:11.509835 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.528273 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.529711 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.531057 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.613047 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.635797 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.647198 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.687750 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.694824 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.698515 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.699862 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.708899 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.712515 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.722730 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.724609 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.765769 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.796500 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.807622 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:11.835701 5974 raft_consensus.cc:482] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:11.836114 5974 raft_consensus.cc:504] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:11.837700 5974 leader_election.cc:288] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:11.840806 4971 leader_election.cc:334] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:11.841599 4974 leader_election.cc:334] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:11.841984 4974 leader_election.cc:302] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:11.842685 5974 raft_consensus.cc:2726] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:11.863897 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.892179 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:11.951016 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.022784 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.027853 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.037420 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.063680 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.064177 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.072516 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:12.115823 5974 raft_consensus.cc:482] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:12.116372 5974 raft_consensus.cc:504] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } } peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } }
I20260810 22:06:12.118441 5974 leader_election.cc:288] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933), 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093)
W20260810 22:06:12.122658 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.123602 4974 leader_election.cc:334] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
W20260810 22:06:12.124120 4971 leader_election.cc:334] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
I20260810 22:06:12.124588 4971 leader_election.cc:302] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:12.125423 5974 raft_consensus.cc:2726] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:12.128301 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.214883 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:12.235960 5974 raft_consensus.cc:482] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:12.236477 5974 raft_consensus.cc:504] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:12.238214 5974 leader_election.cc:288] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:12.241202 4971 leader_election.cc:334] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:12.241824 4974 leader_election.cc:334] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:12.242203 4974 leader_election.cc:302] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:12.242974 5974 raft_consensus.cc:2726] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:12.244869 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.247777 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:12.279381 5974 raft_consensus.cc:482] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:12.279844 5974 raft_consensus.cc:504] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:12.281716 5974 leader_election.cc:288] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:12.284638 4971 leader_election.cc:334] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:12.285025 4974 leader_election.cc:334] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:12.285384 4974 leader_election.cc:302] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:12.286059 5974 raft_consensus.cc:2726] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:12.289165 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.306042 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.338763 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.341861 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.345786 5138 meta_cache.cc:302] tablet 778d8c794af64277b2f497708012e16c: replica b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933) has failed: Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
W20260810 22:06:12.441422 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:12.464210 5974 raft_consensus.cc:482] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:12.464671 5974 raft_consensus.cc:504] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:12.466331 5974 leader_election.cc:288] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:12.469329 4971 leader_election.cc:334] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:12.469851 4974 leader_election.cc:334] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:12.470243 4974 leader_election.cc:302] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
W20260810 22:06:12.470679 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:12.470908 5974 raft_consensus.cc:2726] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:12.489189 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:12.508952 5974 raft_consensus.cc:482] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:12.509479 5974 raft_consensus.cc:504] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:12.511464 5974 leader_election.cc:288] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:12.514273 4971 leader_election.cc:334] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:12.514761 4974 leader_election.cc:334] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:12.515166 4974 leader_election.cc:302] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:12.515880 5974 raft_consensus.cc:2726] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:12.534565 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.576560 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:12.630867 5974 raft_consensus.cc:482] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:12.631268 5974 raft_consensus.cc:504] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:12.632845 5974 leader_election.cc:288] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:12.635907 4971 leader_election.cc:334] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:12.636374 4974 leader_election.cc:334] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:12.636669 4974 leader_election.cc:302] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:12.637228 5974 raft_consensus.cc:2726] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:06:12.637784 5974 raft_consensus.cc:482] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:12.638118 5974 raft_consensus.cc:504] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:12.639755 5974 leader_election.cc:288] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:12.642966 4971 leader_election.cc:334] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:12.644174 4974 leader_election.cc:334] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:12.644524 4974 leader_election.cc:302] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:12.645084 5974 raft_consensus.cc:2726] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:12.660722 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.711464 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.719494 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.724308 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.727003 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.759560 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.776993 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.814729 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.821892 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.856914 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.948030 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:12.962548 5974 raft_consensus.cc:482] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:12.963034 5974 raft_consensus.cc:504] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:12.964761 5974 leader_election.cc:288] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:12.966187 4974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20260810 22:06:12.968837 4971 leader_election.cc:334] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:12.969641 4974 leader_election.cc:334] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:12.970027 4974 leader_election.cc:302] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:12.971060 5974 raft_consensus.cc:2726] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:12.972780 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.973577 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:12.973752 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.002125 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.011706 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.071228 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.072377 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.205610 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.243678 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.337630 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.374424 5138 meta_cache.cc:302] tablet f0037195580e420fad2a77957025fc74: replica b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933) has failed: Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
W20260810 22:06:13.406873 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.410527 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.414463 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.445081 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.475765 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:13.505133 5979 raft_consensus.cc:482] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:13.505692 5979 raft_consensus.cc:504] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:13.507411 5979 leader_election.cc:288] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:13.512764 4971 leader_election.cc:334] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:13.513254 4974 leader_election.cc:334] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:13.513650 4974 leader_election.cc:302] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:13.514346 5979 raft_consensus.cc:2726] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:13.549470 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.616848 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.619053 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.652293 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.682036 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.725538 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.786350 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.867527 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.932480 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:13.968277 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:14.012590 5979 raft_consensus.cc:482] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:14.013025 5979 raft_consensus.cc:504] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:14.014737 5979 leader_election.cc:288] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:14.019345 4971 leader_election.cc:334] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:14.019868 4974 leader_election.cc:334] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:14.020164 4974 leader_election.cc:302] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:14.020741 5979 raft_consensus.cc:2726] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:14.044341 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.045584 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.047583 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.050136 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.079443 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.106109 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.106370 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.109707 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.110889 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.113165 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.116679 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.124747 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.130295 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.144519 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.154318 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.269466 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.269469 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.329545 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.373955 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.375100 5137 meta_cache.cc:302] tablet 518b19decff74b1baf253900da181d46: replica 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093) has failed: Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:14.406093 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.514097 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.552734 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.598694 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.613796 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.623327 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.672958 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.789086 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.789417 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.790464 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.827225 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.907531 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.946642 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:14.946900 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:14.955009 5987 raft_consensus.cc:482] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:14.955504 5987 raft_consensus.cc:504] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:14.957612 5987 leader_election.cc:288] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:14.962515 4971 leader_election.cc:334] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:14.963622 4974 leader_election.cc:334] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:14.963908 4974 leader_election.cc:302] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:14.964476 5987 raft_consensus.cc:2726] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:14.969077 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:15.079608 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:15.175647 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:15.260637 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:15.301397 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:15.324383 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:15.377228 5138 meta_cache.cc:302] tablet 778d8c794af64277b2f497708012e16c: replica b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933) has failed: Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
W20260810 22:06:15.432087 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:15.482818 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:15.493266 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:15.493717 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:15.498199 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:15.535857 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:15.566350 5992 raft_consensus.cc:482] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:15.566756 5992 raft_consensus.cc:504] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:15.568385 5992 leader_election.cc:288] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:15.572894 4971 leader_election.cc:334] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:15.573479 4974 leader_election.cc:334] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:15.573841 4974 leader_election.cc:302] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:15.574405 5992 raft_consensus.cc:2726] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:06:15.714824 5992 raft_consensus.cc:482] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:15.716163 5992 raft_consensus.cc:504] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
W20260810 22:06:15.717888 4999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:15.717837 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:15.719504 5992 leader_election.cc:288] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:15.720902 4994 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:15.717814 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:15.724356 4974 leader_election.cc:334] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
W20260810 22:06:15.726476 4971 leader_election.cc:334] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
I20260810 22:06:15.726763 4971 leader_election.cc:302] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:15.727351 5992 raft_consensus.cc:2726] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:15.788082 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:15.802788 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:15.803164 4994 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:15.805300 4994 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:15.975029 4994 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:15.978394 4994 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.041630 4994 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.041754 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.043119 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.053273 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.058610 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:16.062201 5992 raft_consensus.cc:482] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:16.062839 5992 raft_consensus.cc:504] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:16.066032 5992 leader_election.cc:288] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:16.068624 4971 leader_election.cc:334] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:16.069416 4974 leader_election.cc:334] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:16.069780 4974 leader_election.cc:302] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:16.070405 5992 raft_consensus.cc:2726] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:16.104106 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.235380 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.236171 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.299396 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.307072 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.360906 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.378055 5138 meta_cache.cc:302] tablet 518b19decff74b1baf253900da181d46: replica b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933) has failed: Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
W20260810 22:06:16.487049 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.495694 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.514454 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.559834 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.565743 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.573675 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.639194 5119 debug-util.cc:398] Leaking SignalData structure 0x7b08000e3da0 after lost signal to thread 4945
W20260810 22:06:16.640196 5119 debug-util.cc:398] Leaking SignalData structure 0x7b080005a0a0 after lost signal to thread 5122
I20260810 22:06:16.681996 5998 raft_consensus.cc:482] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:16.682515 5998 raft_consensus.cc:504] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:16.684209 5998 leader_election.cc:288] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:16.690886 4971 leader_election.cc:334] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:16.694697 4974 leader_election.cc:334] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:16.695098 4974 leader_election.cc:302] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:16.695873 5998 raft_consensus.cc:2726] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:06:16.723397 5998 raft_consensus.cc:482] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:16.724061 5998 raft_consensus.cc:504] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:16.726359 5998 leader_election.cc:288] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:16.729864 4971 leader_election.cc:334] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:16.731287 4974 leader_election.cc:334] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:16.731685 4974 leader_election.cc:302] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:16.732429 5998 raft_consensus.cc:2726] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:16.734992 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.748373 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.748799 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.753855 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.789301 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:16.828011 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:17.019788 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:17.028502 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:17.051118 5998 raft_consensus.cc:482] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:17.051692 5998 raft_consensus.cc:504] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } } peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } }
I20260810 22:06:17.053740 5998 leader_election.cc:288] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933), 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093)
W20260810 22:06:17.056927 4971 leader_election.cc:334] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:17.057399 4974 leader_election.cc:334] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:17.057756 4974 leader_election.cc:302] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:17.058444 5998 raft_consensus.cc:2726] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:17.102722 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:17.147544 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:17.156847 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:17.241089 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:17.295230 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:17.312621 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:17.375870 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:17.378844 5137 meta_cache.cc:302] tablet af1fb81326b54dbbb23c7b577bdaf751: replica 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093) has failed: Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:17.405200 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:17.452114 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:17.454550 5998 raft_consensus.cc:482] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:17.455123 5998 raft_consensus.cc:504] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:17.457288 5998 leader_election.cc:288] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:17.460777 4971 leader_election.cc:334] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:17.461189 4974 leader_election.cc:334] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:17.461597 4974 leader_election.cc:302] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:17.462410 5998 raft_consensus.cc:2726] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:17.558843 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:17.656239 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:17.692421 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:17.696328 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:17.727788 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:17.793748 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:17.814164 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:17.834841 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:17.966315 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:17.980134 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:18.075912 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:18.093943 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:18.149681 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:18.165058 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:18.199553 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:18.203984 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:18.220811 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:18.364811 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:18.386500 5138 meta_cache.cc:302] tablet dc4801ac5aeb4b4d8340ee2f5c0d73f6: replica b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933) has failed: Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
W20260810 22:06:18.389412 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:18.457094 6002 raft_consensus.cc:482] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:18.457603 6002 raft_consensus.cc:504] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:18.459507 6002 leader_election.cc:288] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:18.461615 4971 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260810 22:06:18.465524 4971 leader_election.cc:334] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:18.465988 4974 leader_election.cc:334] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:18.466281 4974 leader_election.cc:302] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:18.466848 6002 raft_consensus.cc:2726] T af1fb81326b54dbbb23c7b577bdaf751 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:18.488456 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:18.511168 6002 raft_consensus.cc:482] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:18.511752 6002 raft_consensus.cc:504] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:18.513945 6002 leader_election.cc:288] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:18.517225 4974 leader_election.cc:334] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
W20260810 22:06:18.517866 4971 leader_election.cc:334] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
I20260810 22:06:18.518253 4971 leader_election.cc:302] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:18.518980 6002 raft_consensus.cc:2726] T f0037195580e420fad2a77957025fc74 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:18.529852 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:18.653889 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:18.716023 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:18.820852 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:18.866698 6002 raft_consensus.cc:482] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:18.867272 6002 raft_consensus.cc:504] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:18.868953 6002 leader_election.cc:288] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:18.872116 4971 leader_election.cc:334] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:18.872696 4974 leader_election.cc:334] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:18.873047 4974 leader_election.cc:302] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:18.873806 6002 raft_consensus.cc:2726] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:18.924108 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:18.928278 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:18.959398 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:19.094023 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:19.104214 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:19.152899 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:19.259776 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:19.369741 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:19.387090 5138 meta_cache.cc:302] tablet af1fb81326b54dbbb23c7b577bdaf751: replica b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933) has failed: Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
W20260810 22:06:19.400053 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:19.541582 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:19.550606 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:19.599224 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:19.672473 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:19.711813 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:19.806890 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:19.823597 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:19.850455 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:19.862356 6011 raft_consensus.cc:482] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:19.862897 6011 raft_consensus.cc:504] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:19.865020 6011 leader_election.cc:288] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:19.869645 4971 leader_election.cc:334] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:19.870169 4974 leader_election.cc:334] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:19.870586 4974 leader_election.cc:302] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:19.871165 6011 raft_consensus.cc:2726] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:19.994542 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:20.004163 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:20.056263 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:20.169993 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:20.288297 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:20.297590 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:20.392278 5137 meta_cache.cc:302] tablet 8e619eaa340248da8a54a913c1d16b9f: replica 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093) has failed: Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
I20260810 22:06:20.414314 6017 raft_consensus.cc:482] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:20.414791 6017 raft_consensus.cc:504] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:20.416534 6017 leader_election.cc:288] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:20.421092 4971 leader_election.cc:334] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:20.421613 4974 leader_election.cc:334] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:20.421866 4974 leader_election.cc:302] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:20.422374 6017 raft_consensus.cc:2726] T 628379a2077c4ea7896784b089a5ea0c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:06:20.424152 6017 raft_consensus.cc:482] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:20.424515 6017 raft_consensus.cc:504] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:20.426003 6017 leader_election.cc:288] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:20.428795 4971 leader_election.cc:334] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:20.429215 4974 leader_election.cc:334] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:20.429728 4974 leader_election.cc:302] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:20.430812 6017 raft_consensus.cc:2726] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:20.456292 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:20.458609 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:20.473605 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:20.485988 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:20.488108 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:20.488525 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:20.519970 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:20.530571 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:20.646387 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:20.770524 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:20.782341 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:20.839946 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:20.948132 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:21.068478 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:21.076694 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:21.077945 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:21.096112 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:21.154455 6022 raft_consensus.cc:482] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:21.154958 6022 raft_consensus.cc:504] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:21.157213 6022 leader_election.cc:288] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:21.162845 4971 leader_election.cc:334] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:21.163461 4974 leader_election.cc:334] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:21.163832 4974 leader_election.cc:302] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:21.164520 6022 raft_consensus.cc:2726] T 778d8c794af64277b2f497708012e16c P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:21.250067 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:21.264535 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:21.326824 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:21.398927 5138 meta_cache.cc:302] tablet 628379a2077c4ea7896784b089a5ea0c: replica b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933) has failed: Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
W20260810 22:06:21.428001 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:21.559998 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:21.566593 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:21.584797 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:21.607735 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:21.611645 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:21.633504 6022 raft_consensus.cc:482] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:21.633984 6022 raft_consensus.cc:504] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:21.635594 6022 leader_election.cc:288] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:21.639063 4971 leader_election.cc:334] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:21.639534 4974 leader_election.cc:334] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:21.639840 4974 leader_election.cc:302] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:21.640354 6022 raft_consensus.cc:2726] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:21.740592 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:21.818675 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:21.918383 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:22.055048 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:22.061420 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:22.066953 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:22.071125 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:22.078696 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:22.082175 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:22.120606 6022 raft_consensus.cc:482] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:22.121097 6022 raft_consensus.cc:504] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:22.122925 6022 leader_election.cc:288] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:22.126111 4971 leader_election.cc:334] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:22.126637 4974 leader_election.cc:334] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:22.126940 4974 leader_election.cc:302] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:22.127492 6022 raft_consensus.cc:2726] T 518b19decff74b1baf253900da181d46 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:22.238652 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:22.321866 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:22.401901 5137 meta_cache.cc:302] tablet 778d8c794af64277b2f497708012e16c: replica 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093) has failed: Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:22.418494 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:22.564034 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:22.566349 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:22.581470 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:22.746090 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:22.754869 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:22.757582 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:22.837496 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:22.928690 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:23.082953 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:23.084492 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:23.095220 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:23.115335 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:23.117087 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:23.185194 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:23.264164 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:23.361621 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:23.434826 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:23.437062 5137 meta_cache.cc:302] tablet dc4801ac5aeb4b4d8340ee2f5c0d73f6: replica 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093) has failed: Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:23.446656 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:23.476994 6027 raft_consensus.cc:482] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:23.477458 6027 raft_consensus.cc:504] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:23.479143 6027 leader_election.cc:288] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:23.480980 4971 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20260810 22:06:23.484025 4974 leader_election.cc:334] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
W20260810 22:06:23.484438 4971 leader_election.cc:334] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
I20260810 22:06:23.484766 4971 leader_election.cc:302] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:23.485383 6027 raft_consensus.cc:2726] T 8e619eaa340248da8a54a913c1d16b9f P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:23.712361 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:23.791553 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:23.798393 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:23.882823 6027 raft_consensus.cc:482] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
I20260810 22:06:23.883263 6027 raft_consensus.cc:504] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } } peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } }
I20260810 22:06:23.884728 6027 leader_election.cc:288] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933), 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093)
W20260810 22:06:23.888655 4971 leader_election.cc:334] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:23.889118 4974 leader_election.cc:334] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:23.889523 4974 leader_election.cc:302] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:23.890220 6027 raft_consensus.cc:2726] T dc4801ac5aeb4b4d8340ee2f5c0d73f6 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:23.956601 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:23.977124 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:24.070153 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:24.135188 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:24.156145 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:24.167059 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:24.178304 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:24.314951 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:24.328167 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:24.338995 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:24.444387 5137 meta_cache.cc:302] tablet 416ba6bb2af24654a64eb1a23261254b: replica 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093) has failed: Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:24.495296 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:24.512223 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:24.610227 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:24.684938 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:24.695312 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:24.700350 6035 raft_consensus.cc:482] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9235c169d85402faf437d72e9a38bdd)
W20260810 22:06:24.702659 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:24.703269 6035 raft_consensus.cc:504] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:24.705415 6035 leader_election.cc:288] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:24.728140 4971 leader_election.cc:334] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:24.729434 4974 leader_election.cc:334] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:24.729769 4974 leader_election.cc:302] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:24.730341 6035 raft_consensus.cc:2726] T df71b8fe94f84062bbf858f1cc4a0920 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:24.878309 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:24.881729 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:24.891880 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:24.908425 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:25.050254 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:25.158082 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:25.239058 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:25.251929 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:25.435806 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:25.442276 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:25.462826 5138 meta_cache.cc:302] tablet f0037195580e420fad2a77957025fc74: replica b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933) has failed: Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
W20260810 22:06:25.526922 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:25.599576 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:25.626104 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:25.629431 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:25.712896 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:25.972510 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:26.003376 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:26.184396 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:26.193658 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:26.351788 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:26.365168 5138 batcher.cc:441] Timed out: Failed to write batch of 6 ops to tablet 778d8c794af64277b2f497708012e16c after 186 attempt(s): Failed to write to server: (no server available): Write(tablet: 778d8c794af64277b2f497708012e16c, num_ops: 6, num_attempts: 186) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 22:06:26.466753 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:26.470922 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:26.481488 5137 meta_cache.cc:302] tablet 416ba6bb2af24654a64eb1a23261254b: replica 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093) has failed: Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:26.487258 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:26.559885 4995 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:26.561275 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:26.567564 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:26.581575 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:26.597216 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
W20260810 22:06:26.654078 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
I20260810 22:06:26.839567 6040 raft_consensus.cc:482] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:26.840194 6040 raft_consensus.cc:504] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:26.842236 6040 leader_election.cc:288] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)
W20260810 22:06:26.848625 4971 leader_election.cc:334] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093): Network error: Client connection negotiation failed: client connection to 127.27.17.129:43093: connect: Connection refused (error 111)
W20260810 22:06:26.849229 4974 leader_election.cc:334] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933): Network error: Client connection negotiation failed: client connection to 127.27.17.130:37933: connect: Connection refused (error 111)
I20260810 22:06:26.849597 4974 leader_election.cc:302] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552c4b7a11b4cb7a2fdaacbcdf219dc; no voters: 2b27c8ff42ea450bb5114ab97c1ed25d, b9235c169d85402faf437d72e9a38bdd
I20260810 22:06:26.850127 6040 raft_consensus.cc:2726] T 416ba6bb2af24654a64eb1a23261254b P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260810 22:06:26.866044 4997 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54344: Illegal state: replica a552c4b7a11b4cb7a2fdaacbcdf219dc is not leader of this config: current role FOLLOWER
F20260810 22:06:27.003396 5297 test_workload.cc:301] Timed out: Failed to write batch of 2 ops to tablet dc4801ac5aeb4b4d8340ee2f5c0d73f6 after 187 attempt(s): Failed to write to server: (no server available): Write(tablet: dc4801ac5aeb4b4d8340ee2f5c0d73f6, num_ops: 2, num_attempts: 187) passed its deadline: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
*** Check failure stack trace: ***
*** Aborted at 1786399587 (unix time) try "date -d @1786399587" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800006c46) received by PID 27718 (TID 0x7f5d588f4700) from PID 27718; stack trace: ***
@ 0x2e9ef0 __tsan::CallUserSignalHandler() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2ec5e0 rtl_sigaction() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7f5d67283980 (unknown) at ??:0
@ 0x7f5d62540e87 gsignal at ??:0
@ 0x7f5d625427f1 abort at ??:0
@ 0x2ee337 __interceptor_abort at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f5d65574df4 google::LogMessage::SendToLog() at ??:0
@ 0x7f5d655759a0 google::LogMessage::Flush() at ??:0
@ 0x7f5d6557aadb google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5d6f74f45e kudu::TestWorkload::GetNumberOfErrors() at ??:0
@ 0x7f5d6f74ecd2 kudu::TestWorkload::WriteThread() at ??:0
@ 0x7f5d6f75a78e _ZNSt3__18__invokeIMN4kudu12TestWorkloadEFvvEPS2_JEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS6_DpOS7_ at ??:0
@ 0x7f5d6f75a641 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN4kudu12TestWorkloadEFvvEJPS7_EJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at ??:0
@ 0x7f5d6f759d8b std::__1::__thread_proxy<>() at ??:0
@ 0x2ea5b0 __tsan_thread_start_func at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
@ 0x7f5d672786db start_thread at ??:0
@ 0x7f5d6262361f clone at ??:0
I20260810 22:06:27.229206 6040 raft_consensus.cc:482] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b27c8ff42ea450bb5114ab97c1ed25d)
I20260810 22:06:27.229856 6040 raft_consensus.cc:504] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b27c8ff42ea450bb5114ab97c1ed25d" member_type: VOTER last_known_addr { host: "127.27.17.129" port: 43093 } } peers { permanent_uuid: "b9235c169d85402faf437d72e9a38bdd" member_type: VOTER last_known_addr { host: "127.27.17.130" port: 37933 } } peers { permanent_uuid: "a552c4b7a11b4cb7a2fdaacbcdf219dc" member_type: VOTER last_known_addr { host: "127.27.17.131" port: 44793 } }
I20260810 22:06:27.231961 6040 leader_election.cc:288] T e33e69aaf4a64216b2f995a9e9e9e1e2 P a552c4b7a11b4cb7a2fdaacbcdf219dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b27c8ff42ea450bb5114ab97c1ed25d (127.27.17.129:43093), b9235c169d85402faf437d72e9a38bdd (127.27.17.130:37933)